[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:29 dist-test-slave-2j7r krb5kdc[19736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:29 dist-test-slave-2j7r krb5kdc[19736](info): set up 2 sockets
Aug 10 19:25:29 dist-test-slave-2j7r krb5kdc[19736](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 19:25:31.022698 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:31 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:31.212894 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37947
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:31.542007 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:31.542307 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:31.542416 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:31.549759 19769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:31.549868 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:31.549952 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:31.550029 19769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:31.550098 19769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:31.560199 19769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:31.561573 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:31.562863 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:31.574527 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:31.574575 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:31.574607 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:31.575459 19769 server_base.cc:1034] running on GCE node
I20260810 19:25:31.576243 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:31.577744 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:31.578970 19769 hybrid_clock.cc:648] HybridClock initialized: now 1786389931578926 us; error 100 us; skew 500 ppm
I20260810 19:25:31.579326 19769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:31 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389931, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:31.586181 19769 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:31.587611 19769 webserver.cc:460] Webserver started at http://127.19.68.62:38493/ using document root <none> and password file <none>
I20260810 19:25:31.588272 19769 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:31.588387 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:31.588671 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:31.591746 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "401361e8cf3245eaab95fe698053bdd7"
format_stamp: "Formatted at 2026-08-10 19:25:31 on dist-test-slave-2j7r"
server_key: "5671a73a24f16dec24888435be6d82ca"
server_key_iv: "e0e549440527b67246d6b38f4acae728"
server_key_version: "encryptionkey@0"
I20260810 19:25:31.592470 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "401361e8cf3245eaab95fe698053bdd7"
format_stamp: "Formatted at 2026-08-10 19:25:31 on dist-test-slave-2j7r"
server_key: "5671a73a24f16dec24888435be6d82ca"
server_key_iv: "e0e549440527b67246d6b38f4acae728"
server_key_version: "encryptionkey@0"
I20260810 19:25:31.597416 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:25:31.600631 19795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:31.601919 19769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:31.602066 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "401361e8cf3245eaab95fe698053bdd7"
format_stamp: "Formatted at 2026-08-10 19:25:31 on dist-test-slave-2j7r"
server_key: "5671a73a24f16dec24888435be6d82ca"
server_key_iv: "e0e549440527b67246d6b38f4acae728"
server_key_version: "encryptionkey@0"
I20260810 19:25:31.602229 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:31.622431 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:31.626785 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:31.627049 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:31.645355 19769 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:37947
I20260810 19:25:31.645382 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:37947 every 8 connection(s)
I20260810 19:25:31.646790 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:31.650681 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:31.652166 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19769
I20260810 19:25:31.652483 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:31.654364 19728 external_mini_cluster.cc:1428] Setting key 7c5b8d100edb47c60ea2ae1f9447a8e0
I20260810 19:25:31.662137 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: Bootstrap starting.
I20260810 19:25:31.665026 19898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:31.666095 19898 log.cc:824] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:31.668870 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: No bootstrap required, opened a new log
I20260810 19:25:31.676085 19898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } }
I20260810 19:25:31.676445 19898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:31.676546 19898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401361e8cf3245eaab95fe698053bdd7, State: Initialized, Role: FOLLOWER
I20260810 19:25:31.677282 19898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } }
I20260810 19:25:31.677536 19898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:31.677723 19898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:31.677902 19898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:31.679996 19898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } }
I20260810 19:25:31.680472 19898 leader_election.cc:302] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401361e8cf3245eaab95fe698053bdd7; no voters:
I20260810 19:25:31.681073 19898 leader_election.cc:288] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:31.681210 19907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 19:25:31 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:31.682386 19907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [term 1 LEADER]: Becoming Leader. State: Replica: 401361e8cf3245eaab95fe698053bdd7, State: Running, Role: LEADER
I20260810 19:25:31.682977 19907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } }
I20260810 19:25:31.683339 19898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:31.686056 19915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401361e8cf3245eaab95fe698053bdd7. Latest consensus state: current_term: 1 leader_uuid: "401361e8cf3245eaab95fe698053bdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } } }
I20260810 19:25:31.686810 19915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:31.686811 19912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401361e8cf3245eaab95fe698053bdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401361e8cf3245eaab95fe698053bdd7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37947 } } }
I20260810 19:25:31.687911 19912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:31.698746 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:31.658338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.19.68.62:37947)
0810 19:25:31.659132 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:31.659143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:31.660466 (+ 1323us) server_negotiation.cc:407] Connection header received
0810 19:25:31.669194 (+ 8728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:31.669215 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:31.669577 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:31.669917 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:31.673115 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:31.674563 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:31.676268 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:31.677154 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:31.677371 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:31.684031 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:31.684040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:31.684053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:31.684100 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:31.689856 (+ 5756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:31.691078 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:31.691089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:31.691098 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:31.691383 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:31.692062 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:31.692068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:31.692073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:31.693928 (+ 1855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:31.694322 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:31.697250 (+ 2928us) server_negotiation.cc:299] Negotiation successful
0810 19:25:31.697696 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
W20260810 19:25:31.701400 19937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:25:31.701552 19937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:25:31.701704 19921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:31.703867 19921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:31.712555 19921 catalog_manager.cc:1347] Generated new cluster ID: 6cd3c53103a74762acb55fe47b148c30
I20260810 19:25:31.712658 19921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:31.765494 19921 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:31.767390 19921 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:31.779274 19921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401361e8cf3245eaab95fe698053bdd7: Generated new TSK 0
I20260810 19:25:31.780324 19921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:31.830252 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:32.155956 19948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:32.156221 19948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:32.156317 19948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:32.163421 19948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:32.163525 19948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:32.163652 19948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:25:32.173273 19948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:32.174436 19948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:32.175681 19948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:32.188961 19955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:32.189079 19964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:32.189139 19958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:32.189970 19948 server_base.cc:1034] running on GCE node
I20260810 19:25:32.190523 19948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:32.191377 19948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:32.192581 19948 hybrid_clock.cc:648] HybridClock initialized: now 1786389932192538 us; error 30 us; skew 500 ppm
I20260810 19:25:32.192875 19948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:32 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:32.196071 19948 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:25:32.197711 19948 webserver.cc:460] Webserver started at http://127.19.68.1:42697/ using document root <none> and password file <none>
I20260810 19:25:32.198436 19948 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:32.198541 19948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:32.198799 19948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:32.201854 19948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "06c26d2d6f1e4e1ead54486fc486e7bd"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "5a5f429264daa0157c378f077237438c"
server_key_iv: "efd6f20a7d488ecabf5d224993cc6d52"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.202597 19948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "06c26d2d6f1e4e1ead54486fc486e7bd"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "5a5f429264daa0157c378f077237438c"
server_key_iv: "efd6f20a7d488ecabf5d224993cc6d52"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.207970 19948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:25:32.211555 19982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.212911 19948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:25:32.213056 19948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "06c26d2d6f1e4e1ead54486fc486e7bd"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "5a5f429264daa0157c378f077237438c"
server_key_iv: "efd6f20a7d488ecabf5d224993cc6d52"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.213280 19948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:32.227043 19948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:32.231241 19948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:32.231508 19948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:32.232439 19948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:32.233820 19948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:32.233891 19948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.233980 19948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:32.234032 19948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.271615 19948 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34113
I20260810 19:25:32.271625 20195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34113 every 8 connection(s)
I20260810 19:25:32.272960 19948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:25:32.277827 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19948
I20260810 19:25:32.278040 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:25:32.278666 19728 external_mini_cluster.cc:1428] Setting key 707568b84ef08a3f561da52d581d69a6
Aug 10 19:25:32 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:32.292393 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:32.275480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58865 (local address 127.19.68.62:37947)
0810 19:25:32.275920 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:32.275925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:32.276730 (+ 805us) server_negotiation.cc:407] Connection header received
0810 19:25:32.278964 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:32.278971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:32.279073 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:32.279176 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:32.280821 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.281541 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.283117 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.283546 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.283657 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:32.287028 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:32.287033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:32.287036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:32.287067 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:32.289471 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:32.290133 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:32.290137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:32.290139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:32.290259 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:32.290666 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:32.290671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:32.290674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:32.290994 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:32.291127 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:32.291772 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 19:25:32.292045 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
I20260810 19:25:32.293373 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:32.275876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37947 (local address 127.19.68.1:58865)
0810 19:25:32.276572 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:32.276616 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:32.278585 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:32.279415 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:32.279426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:32.280046 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:32.280664 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:32.280682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.281689 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.281694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:32.282982 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:32.282995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.283755 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.283762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:32.283985 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:32.284602 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:32.284628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:32.286826 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:32.289605 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:32.289614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:32.289628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:32.290022 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:32.290367 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:32.290372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:32.290376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:32.290560 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:32.291119 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:32.291126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:32.291438 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 19:25:32.291663 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:25:32.292150 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":265,"threads_started":1}
I20260810 19:25:32.294971 20196 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37947
I20260810 19:25:32.295246 20196 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:32.295930 20196 heartbeater.cc:511] Master 127.19.68.62:37947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
I20260810 19:25:32.297991 19824 ts_manager.cc:194] Registered new tserver with Master: 06c26d2d6f1e4e1ead54486fc486e7bd (127.19.68.1:34113)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
I20260810 19:25:32.299469 19824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:58865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:25:32.320060 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:32.643146 20206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:32.643448 20206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:32.643587 20206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:32.650887 20206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:32.651013 20206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:32.651177 20206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:25:32.661001 20206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:32.662317 20206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:32.663641 20206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:32.676580 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:32.676683 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:32.676620 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:32.677947 20206 server_base.cc:1034] running on GCE node
I20260810 19:25:32.678568 20206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:32.679543 20206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:32.680763 20206 hybrid_clock.cc:648] HybridClock initialized: now 1786389932680710 us; error 45 us; skew 500 ppm
I20260810 19:25:32.681071 20206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:32 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:32.684259 20206 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:25:32.685818 20206 webserver.cc:460] Webserver started at http://127.19.68.2:33481/ using document root <none> and password file <none>
I20260810 19:25:32.686486 20206 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:32.686589 20206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:32.686862 20206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:32.689800 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "668f324b80d445929a3d2b4d0cd53e5d"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "4174ea41fd8718a9716db23fb43f6d79"
server_key_iv: "6e05bb64faec381abf8b7fcc863d1f26"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.690548 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "668f324b80d445929a3d2b4d0cd53e5d"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "4174ea41fd8718a9716db23fb43f6d79"
server_key_iv: "6e05bb64faec381abf8b7fcc863d1f26"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.695789 20206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:32.699261 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.700546 20206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:32.700697 20206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "668f324b80d445929a3d2b4d0cd53e5d"
format_stamp: "Formatted at 2026-08-10 19:25:32 on dist-test-slave-2j7r"
server_key: "4174ea41fd8718a9716db23fb43f6d79"
server_key_iv: "6e05bb64faec381abf8b7fcc863d1f26"
server_key_version: "encryptionkey@0"
I20260810 19:25:32.700891 20206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:32.718835 20206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:32.722903 20206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:32.723177 20206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:32.724208 20206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:32.725673 20206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:32.725747 20206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.725876 20206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:32.725926 20206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:32.757510 20206 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:37779
I20260810 19:25:32.757562 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:37779 every 8 connection(s)
I20260810 19:25:32.758735 20206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:25:32.766827 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20206
I20260810 19:25:32.767012 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:25:32.767475 19728 external_mini_cluster.cc:1428] Setting key 6b5ec06bd7ad32835b4798159e154753
Aug 10 19:25:32 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:32.778898 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:32.761536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:55439 (local address 127.19.68.62:37947)
0810 19:25:32.761729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:32.761735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:32.762933 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 19:25:32.764733 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:32.764739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:32.764823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:32.764913 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:32.766440 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.767213 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.768860 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.769415 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.769524 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:32.773350 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:32.773362 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:32.773365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:32.773397 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:32.775750 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:32.776571 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:32.776577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:32.776580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:32.776739 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:32.777192 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:32.777197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:32.777200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:32.777545 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:32.777700 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:32.778333 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 19:25:32.778565 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:32.779803 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:32.762018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37947 (local address 127.19.68.2:55439)
0810 19:25:32.762743 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:32.762786 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:32.764475 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:32.765187 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:32.765196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:32.765697 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:32.766258 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:32.766275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.767367 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.767372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:32.768729 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:32.768742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:32.769634 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:32.769642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:32.769864 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:32.770597 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:32.770624 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:32.773127 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:32.775887 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:32.775895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:32.775910 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:32.776451 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:32.776852 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:32.776855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:32.776858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:32.777056 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:32.777664 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:32.777670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:32.777975 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 19:25:32.778201 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 19:25:32.778586 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260810 19:25:32.781177 20438 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37947
I20260810 19:25:32.781538 20438 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:32.782378 20438 heartbeater.cc:511] Master 127.19.68.62:37947 requested a full tablet report, sending...
I20260810 19:25:32.784053 19824 ts_manager.cc:194] Registered new tserver with Master: 668f324b80d445929a3d2b4d0cd53e5d (127.19.68.2:37779)
I20260810 19:25:32.784929 19824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:55439
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:25:32.812275 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:33.145216 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:33.145478 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:33.145615 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:33.152719 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:33.152817 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:33.152971 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:25:33.162568 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:33.163779 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:33.165120 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:33.177892 20456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:33.177860 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:33.177834 20457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:33.178737 20446 server_base.cc:1034] running on GCE node
I20260810 19:25:33.179292 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:33.180172 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:33.181363 20446 hybrid_clock.cc:648] HybridClock initialized: now 1786389933181303 us; error 56 us; skew 500 ppm
I20260810 19:25:33.181659 20446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:33.184711 20446 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:25:33.186169 20446 webserver.cc:460] Webserver started at http://127.19.68.3:46715/ using document root <none> and password file <none>
I20260810 19:25:33.186791 20446 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:33.186914 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:33.187208 20446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:33.190083 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "82e5c9add7d542f08501fe99f167f0d5"
format_stamp: "Formatted at 2026-08-10 19:25:33 on dist-test-slave-2j7r"
server_key: "bf653e902a8b11756b7314e6fc83838f"
server_key_iv: "1553b14dc9ec140aedf09c9bdad7711a"
server_key_version: "encryptionkey@0"
I20260810 19:25:33.190805 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "82e5c9add7d542f08501fe99f167f0d5"
format_stamp: "Formatted at 2026-08-10 19:25:33 on dist-test-slave-2j7r"
server_key: "bf653e902a8b11756b7314e6fc83838f"
server_key_iv: "1553b14dc9ec140aedf09c9bdad7711a"
server_key_version: "encryptionkey@0"
I20260810 19:25:33.195887 20446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:33.199429 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:33.200723 20446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:33.200913 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "82e5c9add7d542f08501fe99f167f0d5"
format_stamp: "Formatted at 2026-08-10 19:25:33 on dist-test-slave-2j7r"
server_key: "bf653e902a8b11756b7314e6fc83838f"
server_key_iv: "1553b14dc9ec140aedf09c9bdad7711a"
server_key_version: "encryptionkey@0"
I20260810 19:25:33.201131 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:33.211665 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:33.215461 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:33.215700 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:33.216655 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:33.218143 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:33.218220 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:33.218312 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:33.218354 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:33.249379 20446 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:46255
I20260810 19:25:33.249452 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:46255 every 8 connection(s)
I20260810 19:25:33.250634 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:25:33.251288 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20446
I20260810 19:25:33.251502 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:25:33.252048 19728 external_mini_cluster.cc:1428] Setting key 954f14ba00a13b5f41593eccd6a9a9a5
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:33.270041 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.253490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:34477 (local address 127.19.68.62:37947)
0810 19:25:33.253683 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.253689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.255125 (+ 1436us) server_negotiation.cc:407] Connection header received
0810 19:25:33.257025 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.257028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.257129 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.257217 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.258598 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.259340 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.260838 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.261309 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.261401 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.264483 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.264488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.264491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.264522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.267258 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.268047 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.268053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.268057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.268203 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.268592 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.268595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.268597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.268853 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.268964 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.269620 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.269777 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:25:33.270961 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.254082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37947 (local address 127.19.68.3:34477)
0810 19:25:33.254941 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:33.255006 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:33.256782 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:33.257393 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:33.257401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:33.257901 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:33.258446 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:33.258463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.259460 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.259464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:33.260727 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:33.260739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.261476 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.261481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:33.261667 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.262233 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:33.262254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:33.264299 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:33.267418 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:33.267426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:33.267441 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:33.267937 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:33.268301 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:33.268304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:33.268307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:33.268513 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:33.268959 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:33.268965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:33.269297 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 19:25:33.269541 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 19:25:33.269904 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":326,"threads_started":1}
I20260810 19:25:33.272310 20666 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37947
I20260810 19:25:33.272622 20666 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:33.273479 20666 heartbeater.cc:511] Master 127.19.68.62:37947 requested a full tablet report, sending...
I20260810 19:25:33.274986 19824 ts_manager.cc:194] Registered new tserver with Master: 82e5c9add7d542f08501fe99f167f0d5 (127.19.68.3:46255)
I20260810 19:25:33.275787 19824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:34477
I20260810 19:25:33.279991 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:25:33.302811 20196 heartbeater.cc:503] Master 127.19.68.62:37947 was elected leader, sending a full tablet report...
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:33.326759 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.312584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.19.68.1:34113)
0810 19:25:33.313264 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.313271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.313318 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:25:33.313372 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.313379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.313578 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.313694 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.315020 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.316036 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.317222 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.317817 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.317918 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.321653 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.321665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.321677 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.321714 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.324440 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.324985 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.324992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.324994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.325148 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.325456 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.325459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.325461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.325787 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.326040 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.326208 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.326406 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":357,"threads_started":1}
W20260810 19:25:33.327996 20037 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46266
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:33.372509 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.359622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.19.68.1:34113)
0810 19:25:33.359877 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.359883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.360050 (+ 167us) server_negotiation.cc:407] Connection header received
0810 19:25:33.360191 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.360195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.360299 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.360384 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.361712 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.362424 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.363578 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.364012 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.364102 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.367698 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.367703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.367706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.367739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.370102 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.370684 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.370689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.370691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.370812 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.371152 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.371157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.371160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.371410 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.371653 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.371924 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.372152 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 19:25:33.374100 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19948
I20260810 19:25:33.387208 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.375009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.19.68.1:34113)
0810 19:25:33.375164 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.375170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.375492 (+ 322us) server_negotiation.cc:407] Connection header received
0810 19:25:33.375637 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.375643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.375703 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.375776 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.376908 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.377656 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.378878 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.379279 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.379382 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.381815 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.381820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.381823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.381859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.384738 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.385397 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.385403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.385406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.385544 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.385879 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.385884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.385887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.386141 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.386368 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.386679 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.386887 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:25:33.387894 20189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:33.470654 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19948
I20260810 19:25:33.497179 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20206
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:25:33.513597 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.498032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.19.68.2:37779)
0810 19:25:33.498622 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.498630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.498657 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:25:33.498737 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.498744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.498950 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.499141 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.500462 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.501685 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.503090 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.503657 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.503830 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.508155 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.508167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.508181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.508221 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.511146 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.511766 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.511773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.511775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.511904 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.512210 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.512214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.512216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.512579 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.512820 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.513059 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.513303 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260810 19:25:33.514815 20425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:33.596551 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20206
I20260810 19:25:33.615641 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20446
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[19736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:25:33.631737 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:33.616678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.19.68.3:46255)
0810 19:25:33.617371 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:33.617380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:33.617414 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:25:33.617476 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:33.617483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:33.617686 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:33.617815 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:33.619075 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.620114 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.621572 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:33.622236 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:33.622374 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:33.626396 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:33.626408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:33.626422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:33.626457 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:33.629299 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.629862 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.629867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.629870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.629980 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:33.630299 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:33.630304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:33.630306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:33.630731 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:33.630996 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:33.631165 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 19:25:33.631434 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":335,"threads_started":1}
I20260810 19:25:33.633047 20657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:33.714587 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20446
I20260810 19:25:33.738157 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19769
I20260810 19:25:33.739398 19886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:33.841464 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19769
2026-08-10T19:25:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4093 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[20699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[20699](info): set up 2 sockets
Aug 10 19:25:33 dist-test-slave-2j7r krb5kdc[20699](info): commencing operation
krb5kdc: starting...
W20260810 19:25:35.084709 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:35 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:35.277688 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33883
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:33883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:35.599831 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:35.600097 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:35.600189 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:35.607164 20726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:35.607259 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:35.607303 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:35.607365 20726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:35.607409 20726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:35.616662 20726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:33883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:35.617831 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:35.619031 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:35.630498 20732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:35.630436 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:35.630467 20733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:35.631531 20726 server_base.cc:1034] running on GCE node
I20260810 19:25:35.632313 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:35.633678 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:35.634882 20726 hybrid_clock.cc:648] HybridClock initialized: now 1786389935634827 us; error 44 us; skew 500 ppm
I20260810 19:25:35.635231 20726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:35 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389935, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:35.638350 20726 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:35.639833 20726 webserver.cc:460] Webserver started at http://127.19.68.62:43905/ using document root <none> and password file <none>
I20260810 19:25:35.640491 20726 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:35.640601 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:35.640880 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:35.643991 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "967f14a51a854d1595bc1dd286357908"
format_stamp: "Formatted at 2026-08-10 19:25:35 on dist-test-slave-2j7r"
server_key: "0370ffc3fb41e07e37ebfa371fbcafd9"
server_key_iv: "f4da955a880da0860c553e0d7ddfabe3"
server_key_version: "encryptionkey@0"
I20260810 19:25:35.644783 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "967f14a51a854d1595bc1dd286357908"
format_stamp: "Formatted at 2026-08-10 19:25:35 on dist-test-slave-2j7r"
server_key: "0370ffc3fb41e07e37ebfa371fbcafd9"
server_key_iv: "f4da955a880da0860c553e0d7ddfabe3"
server_key_version: "encryptionkey@0"
I20260810 19:25:35.650166 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:25:35.653595 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:35.654893 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:35.655071 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "967f14a51a854d1595bc1dd286357908"
format_stamp: "Formatted at 2026-08-10 19:25:35 on dist-test-slave-2j7r"
server_key: "0370ffc3fb41e07e37ebfa371fbcafd9"
server_key_iv: "f4da955a880da0860c553e0d7ddfabe3"
server_key_version: "encryptionkey@0"
I20260810 19:25:35.655270 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:35.699828 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:35.703969 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:35.704465 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:35.721812 20726 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:33883
I20260810 19:25:35.721840 20834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:33883 every 8 connection(s)
I20260810 19:25:35.723189 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:35.726698 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20726
I20260810 19:25:35.726939 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:35.726986 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:35.727810 19728 external_mini_cluster.cc:1428] Setting key 295ad5e9d16bca541dc1d01d359685f3
I20260810 19:25:35.739219 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908: Bootstrap starting.
Aug 10 19:25:35 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:35.742323 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:35.743412 20838 log.cc:824] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:35.746122 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908: No bootstrap required, opened a new log
I20260810 19:25:35.748909 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:35.729934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.19.68.62:33883)
0810 19:25:35.730689 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:35.730700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:35.730740 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:25:35.732226 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:35.732247 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:35.732623 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:35.732983 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:35.734513 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:35.735751 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:35.736976 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:35.737740 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:35.737920 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:35.742000 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:35.742008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:35.742024 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:35.742060 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:35.745008 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:35.745666 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:35.745673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:35.745678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:35.745843 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:35.746231 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:35.746236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:35.746239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:35.746750 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:35.747051 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:35.747573 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 19:25:35.747946 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":333,"threads_started":1}
I20260810 19:25:35.753633 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } }
I20260810 19:25:35.753954 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:35.754056 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967f14a51a854d1595bc1dd286357908, State: Initialized, Role: FOLLOWER
I20260810 19:25:35.754544 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } }
I20260810 19:25:35.754758 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:35.754887 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:35.755031 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:35.757225 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } }
I20260810 19:25:35.757653 20838 leader_election.cc:302] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967f14a51a854d1595bc1dd286357908; no voters:
I20260810 19:25:35.758173 20838 leader_election.cc:288] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:35.758323 20849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:35.759011 20849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [term 1 LEADER]: Becoming Leader. State: Replica: 967f14a51a854d1595bc1dd286357908, State: Running, Role: LEADER
I20260810 19:25:35.759611 20849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } }
I20260810 19:25:35.760136 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:35.763046 20852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967f14a51a854d1595bc1dd286357908. Latest consensus state: current_term: 1 leader_uuid: "967f14a51a854d1595bc1dd286357908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } } }
I20260810 19:25:35.763188 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967f14a51a854d1595bc1dd286357908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f14a51a854d1595bc1dd286357908" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33883 } } }
I20260810 19:25:35.763471 20852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:35.763526 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:35.764472 20858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:35.769941 20858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:35.779798 20858 catalog_manager.cc:1347] Generated new cluster ID: ae45689a1e0f4a50b7c0a103f67ced78
I20260810 19:25:35.779906 20858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:35.805306 20858 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:35.806624 20858 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:35.818828 20858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967f14a51a854d1595bc1dd286357908: Generated new TSK 0
I20260810 19:25:35.819587 20858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:35.870327 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:36.191732 20882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:36.191991 20882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:36.192085 20882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:36.199103 20882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:36.199213 20882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:36.199335 20882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:25:36.209148 20882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:36.210376 20882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:36.211671 20882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:36.224422 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:36.224401 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:36.224370 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:36.225241 20882 server_base.cc:1034] running on GCE node
I20260810 19:25:36.225911 20882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:36.226877 20882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:36.228085 20882 hybrid_clock.cc:648] HybridClock initialized: now 1786389936228023 us; error 55 us; skew 500 ppm
I20260810 19:25:36.228396 20882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:36 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389936, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:36.231505 20882 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:25:36.232956 20882 webserver.cc:460] Webserver started at http://127.19.68.1:36925/ using document root <none> and password file <none>
I20260810 19:25:36.233683 20882 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:36.233803 20882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:36.234100 20882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:36.237012 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "7714985fb98e445ebe4d05e4e67b0541"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "7d9021c4cfef42dd621dee228747c651"
server_key_iv: "878671d2a0f90cdf25dd243947b4785d"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.237840 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "7714985fb98e445ebe4d05e4e67b0541"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "7d9021c4cfef42dd621dee228747c651"
server_key_iv: "878671d2a0f90cdf25dd243947b4785d"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.242911 20882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:25:36.246276 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.247673 20882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:36.247852 20882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "7714985fb98e445ebe4d05e4e67b0541"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "7d9021c4cfef42dd621dee228747c651"
server_key_iv: "878671d2a0f90cdf25dd243947b4785d"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.248035 20882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:36.263451 20882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:36.267308 20882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:36.267565 20882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:36.268482 20882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:36.269840 20882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:36.269918 20882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.270037 20882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:36.270090 20882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.300129 20882 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:41335
I20260810 19:25:36.300177 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:41335 every 8 connection(s)
I20260810 19:25:36.301422 20882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:25:36.309973 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20882
I20260810 19:25:36.310200 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:25:36.310972 19728 external_mini_cluster.cc:1428] Setting key 57ba0beee5c568f74837c408ad6dec7b
Aug 10 19:25:36 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389936, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:36.321929 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:36.303806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:53129 (local address 127.19.68.62:33883)
0810 19:25:36.304241 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:36.304246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:36.305073 (+ 827us) server_negotiation.cc:407] Connection header received
0810 19:25:36.306918 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:36.306924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:36.307004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:36.307096 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:36.308566 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.309339 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.311552 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.312008 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.312149 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:36.316013 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:36.316018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:36.316022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:36.316055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:36.318762 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:36.319570 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:36.319574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:36.319577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:36.319706 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:36.320115 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:36.320119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:36.320122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:36.320476 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:36.320621 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:36.321348 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 19:25:36.321562 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260810 19:25:36.322870 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:36.304226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33883 (local address 127.19.68.1:53129)
0810 19:25:36.304918 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:36.304962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:36.306652 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:36.307333 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:36.307341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:36.307837 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:36.308393 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:36.308410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.309506 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.309512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:36.311403 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:36.311418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.312214 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.312219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:36.312414 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:36.313136 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:36.313163 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:36.315764 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:36.318915 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:36.318923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:36.318937 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:36.319436 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:36.319816 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:36.319821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:36.319824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:36.320018 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:36.320578 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:36.320584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:36.320967 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 19:25:36.321236 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 19:25:36.321629 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":265,"threads_started":1}
I20260810 19:25:36.324322 21093 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33883
I20260810 19:25:36.324594 21093 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:36.325321 21093 heartbeater.cc:511] Master 127.19.68.62:33883 requested a full tablet report, sending...
I20260810 19:25:36.327572 20773 ts_manager.cc:194] Registered new tserver with Master: 7714985fb98e445ebe4d05e4e67b0541 (127.19.68.1:41335)
I20260810 19:25:36.329622 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:53129
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:25:36.355906 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:36.674575 21101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:36.674849 21101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:36.674976 21101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:36.682085 21101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:36.682188 21101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:36.682343 21101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:25:36.691849 21101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:36.693109 21101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:36.694545 21101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:36.707155 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:36.707195 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:36.707219 21112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:36.707921 21101 server_base.cc:1034] running on GCE node
I20260810 19:25:36.708451 21101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:36.709381 21101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:36.710595 21101 hybrid_clock.cc:648] HybridClock initialized: now 1786389936710543 us; error 38 us; skew 500 ppm
I20260810 19:25:36.710887 21101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:36 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389936, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:36.717394 21101 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:25:36.718820 21101 webserver.cc:460] Webserver started at http://127.19.68.2:40097/ using document root <none> and password file <none>
I20260810 19:25:36.719467 21101 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:36.719566 21101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:36.719830 21101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:36.722810 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "fcd1f3a3b607654423fe0ff1faed0b2f"
server_key_iv: "579f3fb796ebe2f1f3e8ffaf555eccd0"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.723565 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "fcd1f3a3b607654423fe0ff1faed0b2f"
server_key_iv: "579f3fb796ebe2f1f3e8ffaf555eccd0"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.728616 21101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:25:36.732007 21126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.733321 21101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:36.733451 21101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8"
format_stamp: "Formatted at 2026-08-10 19:25:36 on dist-test-slave-2j7r"
server_key: "fcd1f3a3b607654423fe0ff1faed0b2f"
server_key_iv: "579f3fb796ebe2f1f3e8ffaf555eccd0"
server_key_version: "encryptionkey@0"
I20260810 19:25:36.733623 21101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:36.758307 21101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:36.762218 21101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:36.762470 21101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:36.763355 21101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:36.764602 21101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:36.764670 21101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.764783 21101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:36.764833 21101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:36.795179 21101 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:33191
I20260810 19:25:36.795228 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:33191 every 8 connection(s)
I20260810 19:25:36.796434 21101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:25:36.802489 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21101
I20260810 19:25:36.802712 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:25:36.803289 19728 external_mini_cluster.cc:1428] Setting key d6fbd9899c2d4f6e09d425dbd0c72105
Aug 10 19:25:36 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389936, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:36.816236 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:36.799011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49361 (local address 127.19.68.62:33883)
0810 19:25:36.799195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:36.799200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:36.800341 (+ 1141us) server_negotiation.cc:407] Connection header received
0810 19:25:36.802244 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:36.802249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:36.802329 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:36.802425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:36.804335 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.805116 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.806651 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.807051 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.807160 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:36.810594 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:36.810599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:36.810602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:36.810636 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:36.813309 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:36.814011 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:36.814015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:36.814018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:36.814161 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:36.814576 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:36.814580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:36.814582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:36.814832 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:36.814981 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:36.815667 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 19:25:36.815908 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:36.817237 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:36.799445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33883 (local address 127.19.68.2:49361)
0810 19:25:36.800144 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:36.800192 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:36.801935 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:36.802735 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:36.802747 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:36.803478 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:36.804154 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:36.804172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.805294 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.805298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:36.806511 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:36.806523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:36.807279 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:36.807288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:36.807502 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:36.808115 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:36.808138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:36.810386 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:36.813451 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:36.813460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:36.813476 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:36.813896 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:36.814267 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:36.814273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:36.814276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:36.814469 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:36.814973 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:36.814981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:36.815294 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 19:25:36.815559 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 19:25:36.816065 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
I20260810 19:25:36.818676 21331 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33883
I20260810 19:25:36.819015 21331 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:36.819900 21331 heartbeater.cc:511] Master 127.19.68.62:33883 requested a full tablet report, sending...
I20260810 19:25:36.821580 20773 ts_manager.cc:194] Registered new tserver with Master: 9cf7f3e3d2ed440e8637eae45a8f80c8 (127.19.68.2:33191)
I20260810 19:25:36.822670 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:49361
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:25:36.847505 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:37.164435 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:37.164695 21338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:37.164803 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:37.171828 21338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:37.171928 21338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:37.172043 21338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:25:37.181387 21338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:37.182497 21338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:37.183782 21338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:37.196050 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:37.196089 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:37.196110 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:37.197072 21338 server_base.cc:1034] running on GCE node
I20260810 19:25:37.197718 21338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:37.199447 21338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:37.200659 21338 hybrid_clock.cc:648] HybridClock initialized: now 1786389937200600 us; error 57 us; skew 500 ppm
I20260810 19:25:37.201007 21338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:37.204311 21338 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:25:37.205844 21338 webserver.cc:460] Webserver started at http://127.19.68.3:41081/ using document root <none> and password file <none>
I20260810 19:25:37.206568 21338 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:37.206686 21338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:37.206995 21338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:37.210052 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "f550ffe1297b493980a9ccf7d6e8bcdb"
format_stamp: "Formatted at 2026-08-10 19:25:37 on dist-test-slave-2j7r"
server_key: "830b56546777eea215c11104b82125ce"
server_key_iv: "b436b22745e3b7dfd762925ae485edfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:37.210875 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "f550ffe1297b493980a9ccf7d6e8bcdb"
format_stamp: "Formatted at 2026-08-10 19:25:37 on dist-test-slave-2j7r"
server_key: "830b56546777eea215c11104b82125ce"
server_key_iv: "b436b22745e3b7dfd762925ae485edfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:37.216213 21338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:37.219751 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:37.221166 21338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:37.221311 21338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "f550ffe1297b493980a9ccf7d6e8bcdb"
format_stamp: "Formatted at 2026-08-10 19:25:37 on dist-test-slave-2j7r"
server_key: "830b56546777eea215c11104b82125ce"
server_key_iv: "b436b22745e3b7dfd762925ae485edfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:37.221488 21338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:37.238737 21338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:37.242986 21338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:37.243228 21338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:37.244148 21338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:37.245544 21338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:37.245620 21338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:37.245728 21338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:37.245775 21338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:37.274778 21338 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42603
I20260810 19:25:37.274887 21560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42603 every 8 connection(s)
I20260810 19:25:37.276039 21338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:25:37.285280 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21338
I20260810 19:25:37.285466 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:25:37.285892 19728 external_mini_cluster.cc:1428] Setting key a9217c7e4d5dc4883feb3b2e920b0fe4
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:37.295439 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.278591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:56241 (local address 127.19.68.62:33883)
0810 19:25:37.278778 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.278789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.279966 (+ 1177us) server_negotiation.cc:407] Connection header received
0810 19:25:37.281886 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.281892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.281973 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.282062 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.283562 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.284318 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.285933 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.286419 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.286534 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.290153 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.290158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.290161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.290194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.292595 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.293294 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.293299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.293301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.293427 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.293839 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.293842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.293844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.294103 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.294234 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.294884 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.295111 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:25:37.296291 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.279062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33883 (local address 127.19.68.3:56241)
0810 19:25:37.279779 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.279828 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.281631 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.282320 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.282329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.282845 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:37.283402 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.283418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.284465 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.284469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.285799 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.285812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.286658 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.286665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.286931 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.287558 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:37.287585 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:37.289958 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:37.292722 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:37.292730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:37.292745 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:37.293179 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:37.293529 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:37.293534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:37.293538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:37.293720 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:37.294225 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:37.294233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:37.294540 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.294764 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.295232 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260810 19:25:37.297462 21564 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33883
I20260810 19:25:37.297732 21564 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:37.298476 21564 heartbeater.cc:511] Master 127.19.68.62:33883 requested a full tablet report, sending...
I20260810 19:25:37.299934 20773 ts_manager.cc:194] Registered new tserver with Master: f550ffe1297b493980a9ccf7d6e8bcdb (127.19.68.3:42603)
I20260810 19:25:37.300962 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:25:37.301025 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:56241
I20260810 19:25:37.319303 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.307792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.19.68.62:33883)
0810 19:25:37.307948 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.307952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.308464 (+ 512us) server_negotiation.cc:407] Connection header received
0810 19:25:37.308565 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.308568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.308632 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.308717 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.310192 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.310909 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.312078 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.312495 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.312588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.314698 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.314703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.314706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.314734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.317138 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.317687 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.317691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.317693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.317820 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.318127 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.318130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.318132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.318442 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.318618 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.318906 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.319059 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:25:37.325270 20773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:37.331916 20773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:37.332448 21093 heartbeater.cc:503] Master 127.19.68.62:33883 was elected leader, sending a full tablet report...
I20260810 19:25:37.364691 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.344400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.19.68.1:41335)
0810 19:25:37.356290 (+ 11890us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.356300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.356330 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:25:37.356425 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.356431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.356652 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.357257 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:37.358085 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.359741 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.362235 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.363599 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.363918 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.364146 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.364229 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.364398 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11701,"thread_start_us":11668,"threads_started":1}
I20260810 19:25:37.364753 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.344003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:41335 (local address 127.0.0.1:60538)
0810 19:25:37.344792 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.344837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.345031 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.356836 (+ 11805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.356844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.356896 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:37.357895 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.357905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.359895 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.359900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.362054 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.362068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.363825 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.363832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.364049 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.364081 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.364154 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.364284 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":377,"spinlock_wait_cycles":6912,"thread_start_us":384,"threads_started":1}
I20260810 19:25:37.365335 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.345492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.19.68.2:33191)
0810 19:25:37.356157 (+ 10665us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.356165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.356193 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:25:37.356308 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.356318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.356598 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.356801 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:37.357626 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.359069 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.361989 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.363453 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.363585 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.363814 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.364419 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.364589 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10511,"thread_start_us":10372,"threads_started":1}
I20260810 19:25:37.365571 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.344563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42603 (local address 127.0.0.1:53724)
0810 19:25:37.354916 (+ 10353us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.354936 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.355077 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.357261 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.357266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.357292 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:37.357881 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.357894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.361170 (+ 3276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.361175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.362868 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.362881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.364349 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.364356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.364516 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.364547 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.364614 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.364732 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10193,"mutex_wait_us":3,"thread_start_us":10049,"threads_started":1}
I20260810 19:25:37.366107 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.345058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.19.68.3:42603)
0810 19:25:37.356702 (+ 11644us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.356709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.356739 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:25:37.356846 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.356853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.357049 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.357244 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:37.358094 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.359721 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.363024 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.364133 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.364885 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.365130 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.365210 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.365808 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11470,"thread_start_us":11301,"threads_started":1}
I20260810 19:25:37.366714 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.345587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33191 (local address 127.0.0.1:57548)
0810 19:25:37.355546 (+ 9959us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.355560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.355654 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.356836 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.356844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.356878 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:37.357389 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.357406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.359184 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.359189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.361466 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.361481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.363649 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.363655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.363896 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.363953 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.364073 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.364264 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9854,"mutex_wait_us":63,"thread_start_us":10078,"threads_started":1}
I20260810 19:25:37.368806 20986 tablet_service.cc:1467] Processing CreateTablet for tablet 30edc4bd645842cc90867daad9a54160 (DEFAULT_TABLE table=test-table [id=7f6e06e4b0574cf0aa5133674751d803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:37.369534 21444 tablet_service.cc:1467] Processing CreateTablet for tablet 30edc4bd645842cc90867daad9a54160 (DEFAULT_TABLE table=test-table [id=7f6e06e4b0574cf0aa5133674751d803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:37.370118 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edc4bd645842cc90867daad9a54160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:37.370086 21217 tablet_service.cc:1467] Processing CreateTablet for tablet 30edc4bd645842cc90867daad9a54160 (DEFAULT_TABLE table=test-table [id=7f6e06e4b0574cf0aa5133674751d803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:37.371201 21444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edc4bd645842cc90867daad9a54160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:37.371407 21217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edc4bd645842cc90867daad9a54160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:37.381373 21589 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Bootstrap starting.
I20260810 19:25:37.384737 21589 tablet_bootstrap.cc:654] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:37.385464 21591 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Bootstrap starting.
I20260810 19:25:37.386278 21589 log.cc:824] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:37.387817 21592 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Bootstrap starting.
I20260810 19:25:37.388954 21591 tablet_bootstrap.cc:654] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:37.390228 21591 log.cc:824] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:37.390683 21589 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: No bootstrap required, opened a new log
I20260810 19:25:37.390975 21589 ts_tablet_manager.cc:1397] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 19:25:37.391323 21592 tablet_bootstrap.cc:654] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:37.392633 21592 log.cc:824] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:37.399338 21591 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: No bootstrap required, opened a new log
I20260810 19:25:37.399595 21591 ts_tablet_manager.cc:1397] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260810 19:25:37.399653 21592 tablet_bootstrap.cc:492] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: No bootstrap required, opened a new log
I20260810 19:25:37.399916 21592 ts_tablet_manager.cc:1397] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260810 19:25:37.400012 21589 raft_consensus.cc:348] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.400453 21589 raft_consensus.cc:374] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:37.400556 21589 raft_consensus.cc:729] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7714985fb98e445ebe4d05e4e67b0541, State: Initialized, Role: FOLLOWER
I20260810 19:25:37.401209 21589 consensus_queue.cc:260] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.402928 21589 ts_tablet_manager.cc:1428] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260810 19:25:37.408798 21591 raft_consensus.cc:348] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.409175 21591 raft_consensus.cc:374] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:37.408954 21592 raft_consensus.cc:348] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.409282 21591 raft_consensus.cc:729] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf7f3e3d2ed440e8637eae45a8f80c8, State: Initialized, Role: FOLLOWER
I20260810 19:25:37.409325 21592 raft_consensus.cc:374] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:37.409421 21592 raft_consensus.cc:729] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f550ffe1297b493980a9ccf7d6e8bcdb, State: Initialized, Role: FOLLOWER
I20260810 19:25:37.409858 21591 consensus_queue.cc:260] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.409927 21592 consensus_queue.cc:260] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.410851 21331 heartbeater.cc:503] Master 127.19.68.62:33883 was elected leader, sending a full tablet report...
I20260810 19:25:37.410894 21564 heartbeater.cc:503] Master 127.19.68.62:33883 was elected leader, sending a full tablet report...
I20260810 19:25:37.411516 21592 ts_tablet_manager.cc:1428] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 19:25:37.411515 21591 ts_tablet_manager.cc:1428] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260810 19:25:37.527637 21566 tablet.cc:2369] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:25:37.548375 21333 tablet.cc:2369] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:25:37.553674 21095 tablet.cc:2369] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:37.640988 21597 raft_consensus.cc:482] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:37.641292 21597 raft_consensus.cc:504] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.643000 21597 leader_election.cc:288] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf7f3e3d2ed440e8637eae45a8f80c8 (127.19.68.2:33191), f550ffe1297b493980a9ccf7d6e8bcdb (127.19.68.3:42603)
I20260810 19:25:37.651140 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.643474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:42593 (local address 127.19.68.2:33191)
0810 19:25:37.643722 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.643729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.644478 (+ 749us) server_negotiation.cc:407] Connection header received
0810 19:25:37.644709 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.644713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.644779 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.644840 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:37.645693 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.646597 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.649443 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.650265 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.650389 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.650467 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.650740 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.650880 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 19:25:37.651230 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.643299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33191 (local address 127.19.68.1:42593)
0810 19:25:37.644355 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.644372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.644534 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.644900 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.644906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.644933 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:37.645537 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.645551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.646749 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.646754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.649273 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.649286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.650450 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.650455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.650574 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.650599 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.650683 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.650813 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":130,"thread_start_us":662,"threads_started":1}
I20260810 19:25:37.652580 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.643324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42603 (local address 127.19.68.1:40177)
0810 19:25:37.644448 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:37.644466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:37.644589 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:37.644945 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:37.644950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:37.644974 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:37.645443 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.645452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.646708 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.646714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.649272 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:37.649286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.651906 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.651913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:37.652073 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.652098 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:25:37.652172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 19:25:37.652260 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":173,"thread_start_us":751,"threads_started":1}
I20260810 19:25:37.652668 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.643589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40177 (local address 127.19.68.3:42603)
0810 19:25:37.643796 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.643802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.644581 (+ 779us) server_negotiation.cc:407] Connection header received
0810 19:25:37.644741 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.644746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.644814 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.644920 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:37.645617 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.646555 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.650549 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.651696 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.651828 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.651940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.652279 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.652423 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:25:37.652531 21250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edc4bd645842cc90867daad9a54160" candidate_uuid: "7714985fb98e445ebe4d05e4e67b0541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" is_pre_election: true
I20260810 19:25:37.653180 21250 raft_consensus.cc:2454] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7714985fb98e445ebe4d05e4e67b0541 in term 0.
I20260810 19:25:37.653692 21477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edc4bd645842cc90867daad9a54160" candidate_uuid: "7714985fb98e445ebe4d05e4e67b0541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" is_pre_election: true
I20260810 19:25:37.653920 20910 leader_election.cc:302] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7714985fb98e445ebe4d05e4e67b0541, 9cf7f3e3d2ed440e8637eae45a8f80c8; no voters:
I20260810 19:25:37.654237 21477 raft_consensus.cc:2454] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7714985fb98e445ebe4d05e4e67b0541 in term 0.
I20260810 19:25:37.654371 21597 raft_consensus.cc:2781] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:37.654525 21597 raft_consensus.cc:482] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:37.654623 21597 raft_consensus.cc:3037] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:37.656816 21597 raft_consensus.cc:504] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.657416 21597 leader_election.cc:288] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf7f3e3d2ed440e8637eae45a8f80c8 (127.19.68.2:33191), f550ffe1297b493980a9ccf7d6e8bcdb (127.19.68.3:42603)
I20260810 19:25:37.657810 21250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edc4bd645842cc90867daad9a54160" candidate_uuid: "7714985fb98e445ebe4d05e4e67b0541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8"
I20260810 19:25:37.657859 21477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edc4bd645842cc90867daad9a54160" candidate_uuid: "7714985fb98e445ebe4d05e4e67b0541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb"
I20260810 19:25:37.657987 21250 raft_consensus.cc:3037] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:37.658022 21477 raft_consensus.cc:3037] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:37.660230 21250 raft_consensus.cc:2454] T 30edc4bd645842cc90867daad9a54160 P 9cf7f3e3d2ed440e8637eae45a8f80c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7714985fb98e445ebe4d05e4e67b0541 in term 1.
I20260810 19:25:37.660488 21477 raft_consensus.cc:2454] T 30edc4bd645842cc90867daad9a54160 P f550ffe1297b493980a9ccf7d6e8bcdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7714985fb98e445ebe4d05e4e67b0541 in term 1.
I20260810 19:25:37.660645 20910 leader_election.cc:302] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7714985fb98e445ebe4d05e4e67b0541, 9cf7f3e3d2ed440e8637eae45a8f80c8; no voters:
I20260810 19:25:37.660940 21597 raft_consensus.cc:2781] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:37.661468 21597 raft_consensus.cc:686] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [term 1 LEADER]: Becoming Leader. State: Replica: 7714985fb98e445ebe4d05e4e67b0541, State: Running, Role: LEADER
I20260810 19:25:37.662000 21597 consensus_queue.cc:237] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } }
I20260810 19:25:37.666486 20771 catalog_manager.cc:5495] T 30edc4bd645842cc90867daad9a54160 P 7714985fb98e445ebe4d05e4e67b0541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7714985fb98e445ebe4d05e4e67b0541 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "7714985fb98e445ebe4d05e4e67b0541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7714985fb98e445ebe4d05e4e67b0541" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 41335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf7f3e3d2ed440e8637eae45a8f80c8" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f550ffe1297b493980a9ccf7d6e8bcdb" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42603 } health_report { overall_health: UNKNOWN } } }
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:37.736927 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.720665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.19.68.1:41335)
0810 19:25:37.720887 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.720893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.721412 (+ 519us) server_negotiation.cc:407] Connection header received
0810 19:25:37.721603 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.721609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.721712 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.721822 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.723285 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.724334 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.725672 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.726384 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.726493 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.730399 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.730418 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.730438 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.730478 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.734093 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.734812 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.734822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.734826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.735013 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.735405 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.735410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.735414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.735832 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.736114 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.736350 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.736614 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:37.784973 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.772178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.19.68.1:41335)
0810 19:25:37.772425 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.772432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.772594 (+ 162us) server_negotiation.cc:407] Connection header received
0810 19:25:37.772797 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.772801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.772871 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.772950 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.774294 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.775023 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.776233 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.776625 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.776713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.780324 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.780328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.780331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.780381 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.782747 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.783254 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.783258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.783260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.783395 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.783828 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.783831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.783833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.784066 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.784270 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.784493 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.784658 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:37.835805 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.822492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.19.68.1:41335)
0810 19:25:37.822722 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.822729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.822946 (+ 217us) server_negotiation.cc:407] Connection header received
0810 19:25:37.823135 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.823140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.823253 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.823346 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.824651 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.825511 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.826867 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.827276 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.827389 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.831108 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.831112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.831115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.831150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.833681 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.834297 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.834300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.834302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.834434 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.834704 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.834707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.834709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.834953 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.835145 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.835377 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.835529 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 19:25:37.837699 20954 tablet_service.cc:2908] Rejecting scan request for tablet 30edc4bd645842cc90867daad9a54160: Uninitialized: clean time has not yet been initialized
I20260810 19:25:37.840420 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20882
I20260810 19:25:37.852421 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.841393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.19.68.1:41335)
0810 19:25:37.841534 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.841546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.841885 (+ 339us) server_negotiation.cc:407] Connection header received
0810 19:25:37.842007 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.842010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.842083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.842159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.843214 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.843912 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.845112 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.845548 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.845660 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.847796 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.847801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.847804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.847833 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.850231 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.850799 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.850802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.850804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.850911 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.851185 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.851189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.851192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.851449 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.851647 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.851935 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.852106 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:25:37.852982 21085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:37.944188 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20882
I20260810 19:25:37.969686 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21101
Aug 10 19:25:37 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:25:37.984161 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:37.970543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.19.68.2:33191)
0810 19:25:37.970708 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:37.970712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:37.970747 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:25:37.970803 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:37.970807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:37.970897 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:37.970974 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:37.972164 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.972899 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.974308 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:37.974695 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:37.974788 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:37.978661 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:37.978675 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:37.978690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:37.978722 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:37.981491 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.982099 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.982107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.982109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.982259 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:37.982651 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:37.982656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:37.982658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:37.983081 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:37.983333 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:37.983660 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 19:25:37.983821 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:25:37.984807 21322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:38.071671 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21101
I20260810 19:25:38.094540 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21338
Aug 10 19:25:38 dist-test-slave-2j7r krb5kdc[20699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:25:38.109440 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:38.095310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.19.68.3:42603)
0810 19:25:38.095473 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:38.095477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:38.095535 (+ 58us) server_negotiation.cc:407] Connection header received
0810 19:25:38.095727 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:38.095731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:38.095797 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:38.095883 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:38.097213 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:38.097928 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:38.099279 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:38.099683 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:38.099776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:38.104064 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:38.104081 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:38.104096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:38.104137 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:38.106811 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:38.107481 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:38.107489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:38.107491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:38.107630 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:38.107973 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:38.107976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:38.107978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:38.108315 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:38.108541 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:38.108784 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:25:38.108946 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:25:38.110069 21552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:38.194814 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21338
I20260810 19:25:38.217200 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20726
I20260810 19:25:38.218493 20825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:38.325685 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20726
2026-08-10T19:25:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4489 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:38 dist-test-slave-2j7r krb5kdc[21639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:38 dist-test-slave-2j7r krb5kdc[21639](info): set up 2 sockets
Aug 10 19:25:38 dist-test-slave-2j7r krb5kdc[21639](info): commencing operation
krb5kdc: starting...
W20260810 19:25:39.548406 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:39 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:39.754396 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37673
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:40.080425 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:40.080703 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:40.080819 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:40.087975 21663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:40.088080 21663 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:25:40.088142 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:40.088243 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:40.088305 21663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:40.088389 21663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:40.098044 21663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:37673
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:40.099299 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:40.100644 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:40.111956 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:40.111992 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:40.112028 21675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:40.113152 21663 server_base.cc:1034] running on GCE node
I20260810 19:25:40.113916 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:40.115247 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:40.116482 21663 hybrid_clock.cc:648] HybridClock initialized: now 1786389940116414 us; error 39 us; skew 500 ppm
I20260810 19:25:40.116799 21663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:40 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:40.119860 21663 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:40.121384 21663 webserver.cc:460] Webserver started at http://127.19.68.62:44895/ using document root <none> and password file <none>
I20260810 19:25:40.122037 21663 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:40.122135 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:40.122450 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:40.125439 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "0159c11056b24df48afeda76edb6f237"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "f7423417f2e771beeb397aef4791f7ef"
server_key_iv: "45eec4227f8556f9d5534e5bfb663521"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.126196 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "0159c11056b24df48afeda76edb6f237"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "f7423417f2e771beeb397aef4791f7ef"
server_key_iv: "45eec4227f8556f9d5534e5bfb663521"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.131309 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 19:25:40.134734 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:40.136067 21663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:25:40.136235 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "0159c11056b24df48afeda76edb6f237"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "f7423417f2e771beeb397aef4791f7ef"
server_key_iv: "45eec4227f8556f9d5534e5bfb663521"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.136439 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:40.153638 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:40.157640 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:40.157888 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:40.178180 21663 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:37673
I20260810 19:25:40.178186 21785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:37673 every 8 connection(s)
I20260810 19:25:40.179457 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:40.182628 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21663
I20260810 19:25:40.182833 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:40.183132 21787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:40.183521 19728 external_mini_cluster.cc:1428] Setting key dd681e3dd8cd5b94c11350c56dbbddc5
I20260810 19:25:40.196796 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Bootstrap starting.
Aug 10 19:25:40 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:40.200575 21787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:40.201954 21787 log.cc:824] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:40.205468 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: No bootstrap required, opened a new log
I20260810 19:25:40.206827 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:40.185375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.19.68.62:37673)
0810 19:25:40.186220 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:40.186235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:40.186284 (+ 49us) server_negotiation.cc:407] Connection header received
0810 19:25:40.187932 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:40.187959 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:40.188428 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:40.188853 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:40.190113 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.192064 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.193238 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.194274 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.194470 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:40.198388 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:40.198399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:40.198417 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:40.198462 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:40.202194 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.203093 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.203100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.203106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.203304 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.203676 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.203681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.203684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.204398 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:40.204686 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:40.205008 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 19:25:40.205468 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":349,"threads_started":1}
I20260810 19:25:40.213248 21787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } }
I20260810 19:25:40.213603 21787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:40.213676 21787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0159c11056b24df48afeda76edb6f237, State: Initialized, Role: FOLLOWER
I20260810 19:25:40.214167 21787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } }
I20260810 19:25:40.214385 21787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:40.214479 21787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:40.214594 21787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:40.216692 21787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } }
I20260810 19:25:40.217259 21787 leader_election.cc:302] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0159c11056b24df48afeda76edb6f237; no voters:
I20260810 19:25:40.217801 21787 leader_election.cc:288] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:40.217931 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:40.218556 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [term 1 LEADER]: Becoming Leader. State: Replica: 0159c11056b24df48afeda76edb6f237, State: Running, Role: LEADER
I20260810 19:25:40.219054 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } }
I20260810 19:25:40.219830 21787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:40.221745 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0159c11056b24df48afeda76edb6f237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } } }
I20260810 19:25:40.222139 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:40.225593 21812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:40.225543 21802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0159c11056b24df48afeda76edb6f237. Latest consensus state: current_term: 1 leader_uuid: "0159c11056b24df48afeda76edb6f237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0159c11056b24df48afeda76edb6f237" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37673 } } }
I20260810 19:25:40.225878 21802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:40.230633 21812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:40.239981 21812 catalog_manager.cc:1347] Generated new cluster ID: fb2e525d3cf14a85ba83cc8d0567598f
I20260810 19:25:40.240097 21812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:40.257885 21812 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:40.259826 21812 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:40.269876 21812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Generated new TSK 0
I20260810 19:25:40.270929 21812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:40.327107 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37673
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:40.668799 21832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:40.669152 21832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:40.669333 21832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:40.676815 21832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:40.676925 21832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:40.677055 21832 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:40.677198 21832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:25:40.687157 21832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:40.688481 21832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:40.689944 21832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:40.703240 21839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:40.703269 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:40.703374 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:40.704098 21832 server_base.cc:1034] running on GCE node
I20260810 19:25:40.704715 21832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:40.705739 21832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:40.706979 21832 hybrid_clock.cc:648] HybridClock initialized: now 1786389940706914 us; error 36 us; skew 500 ppm
I20260810 19:25:40.707276 21832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:40 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:40.710533 21832 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:25:40.711855 21832 webserver.cc:460] Webserver started at http://127.19.68.1:32881/ using document root <none> and password file <none>
I20260810 19:25:40.712498 21832 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:40.712596 21832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:40.712899 21832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:40.715723 21832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "4df5b534adfb4e929abae771318a2934"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "40721bf48fd5f1f21ed5a9650a41d6cb"
server_key_iv: "8c0fb90b43ea675e4752799efd521543"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.716495 21832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "4df5b534adfb4e929abae771318a2934"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "40721bf48fd5f1f21ed5a9650a41d6cb"
server_key_iv: "8c0fb90b43ea675e4752799efd521543"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.721689 21832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:25:40.725021 21855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:40.726365 21832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:40.726507 21832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "4df5b534adfb4e929abae771318a2934"
format_stamp: "Formatted at 2026-08-10 19:25:40 on dist-test-slave-2j7r"
server_key: "40721bf48fd5f1f21ed5a9650a41d6cb"
server_key_iv: "8c0fb90b43ea675e4752799efd521543"
server_key_version: "encryptionkey@0"
I20260810 19:25:40.726706 21832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:40.741261 21832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:40.745333 21832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:40.745594 21832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:40.748340 21832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:40.748450 21832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:40.748601 21832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:40.748687 21832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:25:40 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:40.771412 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:40.748864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:36303 (local address 127.19.68.62:37673)
0810 19:25:40.749613 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:40.749618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:40.750504 (+ 886us) server_negotiation.cc:407] Connection header received
0810 19:25:40.752522 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:40.752528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:40.752612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:40.752743 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:40.755120 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.756054 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.758439 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.759090 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.759238 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:40.764401 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:40.764407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:40.764412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:40.764456 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:40.767871 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.768881 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.768887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.768891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.769019 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.769591 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.769595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.769597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.769930 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:40.770101 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:40.770819 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 19:25:40.771041 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":544,"threads_started":1}
I20260810 19:25:40.772455 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:40.749511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.1:36303)
0810 19:25:40.750354 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:40.750392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:40.752260 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:40.753030 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:40.753047 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:40.753889 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:40.754899 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:40.754932 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.756268 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.756272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:40.758289 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:40.758302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.759628 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.759634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:40.759831 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:40.760679 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:40.760702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:40.764058 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:40.768036 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:40.768043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:40.768057 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:40.768762 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:40.769149 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:40.769154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:40.769158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:40.769467 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:40.770092 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:40.770100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:40.770452 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 19:25:40.770684 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 19:25:40.771195 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
I20260810 19:25:40.791620 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:40.778202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:54659 (local address 127.19.68.62:37673)
0810 19:25:40.778340 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:40.778343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:40.779155 (+ 812us) server_negotiation.cc:407] Connection header received
0810 19:25:40.779275 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:40.779281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:40.779363 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:40.779471 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:40.780294 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.781447 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.783226 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.783903 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.784029 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:40.786037 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:40.786042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:40.786045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:40.786075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:40.788937 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.789790 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.789796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.789800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.789932 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:40.790278 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:40.790283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:40.790287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:40.790540 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:40.790683 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:40.791077 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 19:25:40.791284 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:25:40.791710 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:40.778419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.1:54659)
0810 19:25:40.779064 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:40.779083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:40.779189 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:40.779481 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:40.779484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:40.779896 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:40.780176 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:40.780184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.781620 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.781625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:40.783074 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:40.783093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:40.784104 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:40.784109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:40.784236 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:40.784850 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:40.784866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:40.785863 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:40.789115 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:40.789121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:40.789125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:40.789679 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:40.790042 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:40.790047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:40.790051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:40.790179 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:40.790662 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:40.790667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:40.790844 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 19:25:40.790959 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:25:40.791284 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260810 19:25:40.800148 21832 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34159
I20260810 19:25:40.800170 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34159 every 8 connection(s)
I20260810 19:25:40.801380 21832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:25:40.803409 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21832
I20260810 19:25:40.803557 22051 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37673
I20260810 19:25:40.803615 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:25:40.803751 22051 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:40.804273 19728 external_mini_cluster.cc:1428] Setting key 6a5831dea5ffdbd834ff834f206bfce1
I20260810 19:25:40.804368 22051 heartbeater.cc:511] Master 127.19.68.62:37673 requested a full tablet report, sending...
I20260810 19:25:40.806432 21712 ts_manager.cc:194] Registered new tserver with Master: 4df5b534adfb4e929abae771318a2934 (127.19.68.1:34159)
I20260810 19:25:40.807832 21712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:36303
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:25:40.848543 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37673
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:41.170186 22057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:41.170506 22057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:41.170608 22057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:41.177650 22057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:41.177762 22057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:41.177889 22057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:41.178005 22057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:25:41.187515 22057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:41.188722 22057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:41.190029 22057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:41.202678 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:41.202646 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:41.202755 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:41.203349 22057 server_base.cc:1034] running on GCE node
I20260810 19:25:41.203977 22057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:41.204921 22057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:41.206166 22057 hybrid_clock.cc:648] HybridClock initialized: now 1786389941206078 us; error 57 us; skew 500 ppm
I20260810 19:25:41.206478 22057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:41.209586 22057 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:25:41.211127 22057 webserver.cc:460] Webserver started at http://127.19.68.2:40719/ using document root <none> and password file <none>
I20260810 19:25:41.211814 22057 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:41.211920 22057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:41.212251 22057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:41.215258 22057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "8fccd521254b453984779baf1c489481"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "f8ecb1f4f57f174bcff7573e640f402c"
server_key_iv: "7d840b1fe5accae48a7941a1b67e69f2"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.216041 22057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "8fccd521254b453984779baf1c489481"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "f8ecb1f4f57f174bcff7573e640f402c"
server_key_iv: "7d840b1fe5accae48a7941a1b67e69f2"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.221171 22057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:25:41.224498 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:41.225839 22057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:41.226006 22057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "8fccd521254b453984779baf1c489481"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "f8ecb1f4f57f174bcff7573e640f402c"
server_key_iv: "7d840b1fe5accae48a7941a1b67e69f2"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.226214 22057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:41.244475 22057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:41.248802 22057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:41.249125 22057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:41.251873 22057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:41.251963 22057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:41.252125 22057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:41.252243 22057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:41.273066 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.252231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:42311 (local address 127.19.68.62:37673)
0810 19:25:41.252441 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.252447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.253816 (+ 1369us) server_negotiation.cc:407] Connection header received
0810 19:25:41.255825 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.255831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.255971 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.256099 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:41.258354 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.259339 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.261770 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.262199 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.262312 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.265908 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:41.265913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:41.265917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:41.265971 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:41.269058 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.269945 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.269951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.269956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.270172 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.270818 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.270824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.270827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.271237 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:41.271399 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.272416 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.272678 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:25:41.274636 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.252733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.2:42311)
0810 19:25:41.253625 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.253669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.255517 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.256419 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.256431 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.257328 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:41.258156 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.258190 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.259519 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.259526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.261626 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.261642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.262440 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.262449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.262649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.263214 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:41.263236 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:41.265641 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:41.269218 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.269227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.269242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.269801 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.270328 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.270334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.270338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.270662 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.271388 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:41.271397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:41.271903 (+ 506us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.272275 (+ 372us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.272877 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":387,"threads_started":1}
I20260810 19:25:41.292006 22057 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:44467
I20260810 19:25:41.292066 22285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:44467 every 8 connection(s)
I20260810 19:25:41.293542 22057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:25:41.295586 22286 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37673
I20260810 19:25:41.295826 22286 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:41.296530 22286 heartbeater.cc:511] Master 127.19.68.62:37673 requested a full tablet report, sending...
I20260810 19:25:41.296573 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22057
I20260810 19:25:41.296788 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:25:41.297381 19728 external_mini_cluster.cc:1428] Setting key d2c69bdedf553d61e5dd7d144e256a06
I20260810 19:25:41.297984 21712 ts_manager.cc:194] Registered new tserver with Master: 8fccd521254b453984779baf1c489481 (127.19.68.2:44467)
I20260810 19:25:41.299086 21712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:42311
I20260810 19:25:41.303117 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.282398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.2:54251)
0810 19:25:41.289663 (+ 7265us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.289685 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.289839 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.290187 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.290193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.290663 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:41.291063 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.291081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.292508 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.292514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.293996 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.294012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.295129 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.295134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.295256 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.295848 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:41.295866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:41.296847 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:41.300575 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.300581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.300585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.301143 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.301544 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.301549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.301553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.301708 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.302359 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:41.302363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:41.302507 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.302606 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.302822 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7096,"thread_start_us":6892,"threads_started":1}
I20260810 19:25:41.303270 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.282028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:54251 (local address 127.19.68.62:37673)
0810 19:25:41.282189 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.282194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.289780 (+ 7586us) server_negotiation.cc:407] Connection header received
0810 19:25:41.289967 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.289971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.290033 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.290123 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:41.291264 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.292373 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.294158 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.294753 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.294882 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.297049 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:41.297055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:41.297060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:41.297120 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:41.300438 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.301263 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.301268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.301271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.301426 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.301830 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.301835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.301838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.302251 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:41.302390 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.302735 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.302978 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:25:41.342051 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37673
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:41.659197 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:41.659499 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:41.659646 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:41.666929 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:41.667053 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:41.667177 22294 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:41.667301 22294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:25:41.677032 22294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:41.678685 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:41.680109 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:41.693414 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:41.693384 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:41.693449 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:41.694527 22294 server_base.cc:1034] running on GCE node
I20260810 19:25:41.695140 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:41.696135 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:41.697377 22294 hybrid_clock.cc:648] HybridClock initialized: now 1786389941697296 us; error 44 us; skew 500 ppm
I20260810 19:25:41.697679 22294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:41.701063 22294 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:25:41.702651 22294 webserver.cc:460] Webserver started at http://127.19.68.3:42749/ using document root <none> and password file <none>
I20260810 19:25:41.703366 22294 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:41.703480 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:41.703822 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:41.707269 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "7038d62fd1ec45bcbb0181a799d46668"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "0f91fe3c6b9a126cd3b1f7ef40940506"
server_key_iv: "8ac7fccb0dd4c22223069757ea2d73b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.708115 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "7038d62fd1ec45bcbb0181a799d46668"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "0f91fe3c6b9a126cd3b1f7ef40940506"
server_key_iv: "8ac7fccb0dd4c22223069757ea2d73b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.713536 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:25:41.716813 22325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:41.718149 22294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:41.718410 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "7038d62fd1ec45bcbb0181a799d46668"
format_stamp: "Formatted at 2026-08-10 19:25:41 on dist-test-slave-2j7r"
server_key: "0f91fe3c6b9a126cd3b1f7ef40940506"
server_key_iv: "8ac7fccb0dd4c22223069757ea2d73b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:41.718627 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:41.747925 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:41.751924 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:41.752184 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:41.754917 22294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:41.755024 22294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:41.755184 22294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:41.755259 22294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:41.777803 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.755341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:54317 (local address 127.19.68.62:37673)
0810 19:25:41.755522 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.755529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.756951 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 19:25:41.759003 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.759009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.759094 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.759181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:41.761196 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.762210 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.765022 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.765589 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.765737 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.770405 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:41.770431 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:41.770437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:41.770481 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:41.773524 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.774496 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.774501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.774505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.774688 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.775361 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.775367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.775370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.775793 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:41.776850 (+ 1057us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.777058 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.777358 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:25:41.778797 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.756000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.3:54317)
0810 19:25:41.756783 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.756824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.758712 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.759442 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.759454 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.760145 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:41.761002 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.761033 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.762413 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.762419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.764845 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.764862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.765935 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.765941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.766232 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.766890 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:41.766915 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:41.770126 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:41.773691 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.773698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.773710 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.774367 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.774856 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.774862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.774866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.775217 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.775984 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:41.775993 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:41.776469 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.776779 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.777355 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
I20260810 19:25:41.796671 22294 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:34347
I20260810 19:25:41.796694 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:34347 every 8 connection(s)
I20260810 19:25:41.797994 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:25:41.800133 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22294
I20260810 19:25:41.800340 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:25:41.800691 22531 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37673
I20260810 19:25:41.800902 19728 external_mini_cluster.cc:1428] Setting key 25bbd41641b03846f99bddc56abe2f2c
I20260810 19:25:41.801007 22531 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:41.801761 22531 heartbeater.cc:511] Master 127.19.68.62:37673 requested a full tablet report, sending...
I20260810 19:25:41.803094 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.788505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.19.68.3:58707)
0810 19:25:41.789211 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.789234 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.789374 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.789690 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.789694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.790388 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:41.790856 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.790868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.792103 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.792108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.793683 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.793699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.794789 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.794795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.794969 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.795550 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:41.795571 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:41.796460 (+ 889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:41.799855 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.799859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.799863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.800367 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.800950 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:41.800955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:41.800958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:41.801160 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:41.801981 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:41.801986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:41.802176 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.802330 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.802646 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":381,"threads_started":1}
I20260810 19:25:41.803802 21714 ts_manager.cc:194] Registered new tserver with Master: 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:41.804232 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.788408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:58707 (local address 127.19.68.62:37673)
0810 19:25:41.788565 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.788568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.789317 (+ 749us) server_negotiation.cc:407] Connection header received
0810 19:25:41.789480 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.789485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.789562 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.789644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:41.791045 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.791902 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.793887 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.794438 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.794511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.796645 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:41.796649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:41.796653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:41.796687 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:41.799661 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.800644 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.800650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.800654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.800816 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.801297 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.801303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.801306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.801875 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:41.802016 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.803628 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.803876 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:25:41.804979 21714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:54317
I20260810 19:25:41.810380 22051 heartbeater.cc:503] Master 127.19.68.62:37673 was elected leader, sending a full tablet report...
I20260810 19:25:41.816076 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:25:41 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:41.865772 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.852562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.19.68.62:37673)
0810 19:25:41.852782 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.852789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.853106 (+ 317us) server_negotiation.cc:407] Connection header received
0810 19:25:41.853295 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.853299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.853376 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.853464 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:41.854811 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.855548 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.856817 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.857277 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.857398 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.860973 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:41.860978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:41.860981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:41.861016 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:41.863319 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.863896 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.863902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.863906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.864030 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:41.864397 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:41.864402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:41.864405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:41.864657 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:41.864894 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.865200 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.865437 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:41.869664 21714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:41.876433 21714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:41.901302 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.888093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:34347 (local address 127.0.0.1:44656)
0810 19:25:41.891134 (+ 3041us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.891232 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.891478 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.892568 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.892573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.892600 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:41.893064 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.893075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.895238 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.895244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.897971 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.897989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.900235 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.900241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.900453 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.900505 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.900640 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.900824 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2862,"mutex_wait_us":91,"thread_start_us":2710,"threads_started":1}
I20260810 19:25:41.902120 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.888644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.19.68.3:34347)
0810 19:25:41.891630 (+ 2986us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.891638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.891673 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:25:41.891769 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.891776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.892020 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.892202 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:41.893310 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.895066 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.898176 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.899711 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.899843 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.900102 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.901573 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.901765 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2816,"thread_start_us":2626,"threads_started":1}
I20260810 19:25:41.903380 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.888005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.19.68.2:44467)
0810 19:25:41.888534 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.888544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.889842 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 19:25:41.890274 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.890283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.890552 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.890715 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:41.892185 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.895073 (+ 2888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.899059 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.900506 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.900611 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.900820 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.902918 (+ 2098us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.903104 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260810 19:25:41.903950 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.887806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44467 (local address 127.0.0.1:42650)
0810 19:25:41.889685 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.889734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.890023 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.891012 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.891018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.891049 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:41.891915 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.891930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.895273 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.895277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.898863 (+ 3586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.898880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.902475 (+ 3595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.902482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.902680 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.902714 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.903432 (+ 718us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.903697 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1604,"mutex_wait_us":71,"spinlock_wait_cycles":432512,"thread_start_us":1464,"threads_started":1}
I20260810 19:25:41.906800 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 66f1949d37af44a9abe5155f2d42636c (DEFAULT_TABLE table=test-table [id=d40e294540e44346bba49029fc9bee3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:41.907310 22159 tablet_service.cc:1467] Processing CreateTablet for tablet 66f1949d37af44a9abe5155f2d42636c (DEFAULT_TABLE table=test-table [id=d40e294540e44346bba49029fc9bee3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:41.908260 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f1949d37af44a9abe5155f2d42636c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:41.908560 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f1949d37af44a9abe5155f2d42636c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:41.910784 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.891463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34159 (local address 127.0.0.1:35858)
0810 19:25:41.902546 (+ 11083us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:41.902566 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:41.902670 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:41.904133 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:41.904138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:41.904164 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:41.904310 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.904320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.906204 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.906209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.907999 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:41.908014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.909756 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.909762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:41.910261 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.910294 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:25:41.910391 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:25:41.910517 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10985,"thread_start_us":11486,"threads_started":1}
I20260810 19:25:41.910892 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:41.891660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.19.68.1:34159)
0810 19:25:41.903553 (+ 11893us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:41.903562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:41.903590 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:25:41.903676 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:41.903683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:41.903923 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:41.904449 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:41.904603 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.906043 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.908159 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:41.909585 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:41.909717 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:41.909916 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:41.910459 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 19:25:41.910618 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11650,"thread_start_us":11519,"threads_started":1}
I20260810 19:25:41.914158 21933 tablet_service.cc:1467] Processing CreateTablet for tablet 66f1949d37af44a9abe5155f2d42636c (DEFAULT_TABLE table=test-table [id=d40e294540e44346bba49029fc9bee3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:41.915284 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f1949d37af44a9abe5155f2d42636c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:41.922493 22563 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Bootstrap starting.
I20260810 19:25:41.924521 22565 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Bootstrap starting.
I20260810 19:25:41.926113 22563 tablet_bootstrap.cc:654] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:41.927271 22566 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Bootstrap starting.
I20260810 19:25:41.927472 22563 log.cc:824] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:41.927769 22565 tablet_bootstrap.cc:654] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:41.928982 22565 log.cc:824] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:41.930739 22566 tablet_bootstrap.cc:654] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:41.931947 22566 log.cc:824] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:41.932331 22565 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: No bootstrap required, opened a new log
I20260810 19:25:41.932621 22565 ts_tablet_manager.cc:1397] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 19:25:41.935006 22566 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: No bootstrap required, opened a new log
I20260810 19:25:41.935083 22563 tablet_bootstrap.cc:492] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: No bootstrap required, opened a new log
I20260810 19:25:41.935298 22566 ts_tablet_manager.cc:1397] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 19:25:41.935313 22563 ts_tablet_manager.cc:1397] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260810 19:25:41.940299 22565 raft_consensus.cc:348] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.940682 22565 raft_consensus.cc:374] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:41.940773 22565 raft_consensus.cc:729] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7038d62fd1ec45bcbb0181a799d46668, State: Initialized, Role: FOLLOWER
I20260810 19:25:41.941346 22565 consensus_queue.cc:260] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.942404 22531 heartbeater.cc:503] Master 127.19.68.62:37673 was elected leader, sending a full tablet report...
I20260810 19:25:41.942914 22565 ts_tablet_manager.cc:1428] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260810 19:25:41.945649 22566 raft_consensus.cc:348] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.946110 22566 raft_consensus.cc:374] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:41.946203 22566 raft_consensus.cc:729] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Initialized, Role: FOLLOWER
I20260810 19:25:41.946810 22566 consensus_queue.cc:260] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.947315 22563 raft_consensus.cc:348] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.947742 22563 raft_consensus.cc:374] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:41.947849 22563 raft_consensus.cc:729] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fccd521254b453984779baf1c489481, State: Initialized, Role: FOLLOWER
I20260810 19:25:41.948424 22563 consensus_queue.cc:260] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:41.949774 22566 ts_tablet_manager.cc:1428] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260810 19:25:41.950060 22563 ts_tablet_manager.cc:1428] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 19:25:41.950415 22286 heartbeater.cc:503] Master 127.19.68.62:37673 was elected leader, sending a full tablet report...
W20260810 19:25:42.045686 22287 tablet.cc:2369] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:25:42.049758 22532 tablet.cc:2369] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:25:42.054036 22052 tablet.cc:2369] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:42.179770 22578 raft_consensus.cc:482] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:42.180114 22578 raft_consensus.cc:504] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:42.182336 22578 leader_election.cc:288] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.190160 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.183244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:43781 (local address 127.19.68.3:34347)
0810 19:25:42.183463 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.183470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.184112 (+ 642us) server_negotiation.cc:407] Connection header received
0810 19:25:42.184303 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.184309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.184390 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.184479 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:42.185247 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.185995 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.187956 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.189192 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.189332 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.189438 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.189787 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.189914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 19:25:42.190194 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.182724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:34347 (local address 127.19.68.1:43781)
0810 19:25:42.184004 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:42.184020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:42.184159 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:42.184665 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:42.184670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:42.184702 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:42.185075 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.185109 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.186133 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.186138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.187796 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.187810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.189434 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.189441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.189622 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.189657 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:25:42.189751 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 19:25:42.189874 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"thread_start_us":916,"threads_started":1}
I20260810 19:25:42.191980 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.182758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44467 (local address 127.19.68.1:59291)
0810 19:25:42.184703 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:42.184719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:42.184853 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:42.186146 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:42.186150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:42.186177 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:42.186626 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.186639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.188026 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.188031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.189798 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.189813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.191299 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.191307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.191463 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.191496 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:25:42.191573 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 19:25:42.191684 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"thread_start_us":1646,"threads_started":1}
I20260810 19:25:42.191738 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f1949d37af44a9abe5155f2d42636c" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668" is_pre_election: true
I20260810 19:25:42.192341 22448 raft_consensus.cc:2454] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.192729 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.182749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:59291 (local address 127.19.68.2:44467)
0810 19:25:42.183231 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.183237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.184812 (+ 1575us) server_negotiation.cc:407] Connection header received
0810 19:25:42.185004 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.185009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.185106 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.185192 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:42.186810 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.187914 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.189978 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.191081 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.192172 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.192297 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.192355 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.192480 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260810 19:25:42.193005 21859 leader_election.cc:302] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 7038d62fd1ec45bcbb0181a799d46668; no voters:
I20260810 19:25:42.193430 22578 raft_consensus.cc:2781] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:42.193596 22578 raft_consensus.cc:482] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:42.193707 22578 raft_consensus.cc:3037] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.193698 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f1949d37af44a9abe5155f2d42636c" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481" is_pre_election: true
I20260810 19:25:42.194244 22193 raft_consensus.cc:2454] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.196954 22578 raft_consensus.cc:504] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:42.197664 22578 leader_election.cc:288] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Requested vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.198125 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f1949d37af44a9abe5155f2d42636c" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481"
I20260810 19:25:42.198149 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f1949d37af44a9abe5155f2d42636c" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668"
I20260810 19:25:42.198343 22193 raft_consensus.cc:3037] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.198396 22448 raft_consensus.cc:3037] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.200594 22448 raft_consensus.cc:2454] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.200598 22193 raft_consensus.cc:2454] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.200992 21859 leader_election.cc:302] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 7038d62fd1ec45bcbb0181a799d46668; no voters:
I20260810 19:25:42.201294 22578 raft_consensus.cc:2781] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:42.201761 22578 raft_consensus.cc:686] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 1 LEADER]: Becoming Leader. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Running, Role: LEADER
I20260810 19:25:42.202301 22578 consensus_queue.cc:237] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } }
I20260810 19:25:42.207175 21714 catalog_manager.cc:5495] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df5b534adfb4e929abae771318a2934 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } health_report { overall_health: HEALTHY } } }
I20260810 19:25:42.270323 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.265061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.19.68.1:34159)
0810 19:25:42.265270 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.265274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.265429 (+ 155us) server_negotiation.cc:407] Connection header received
0810 19:25:42.265683 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.265687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.265801 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.265931 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:42.266328 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.267014 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.268308 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.268717 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.268809 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.269315 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:42.269809 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:42.269913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.270020 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.270089 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:25:42.278132 22193 raft_consensus.cc:1261] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Refusing update from remote peer 4df5b534adfb4e929abae771318a2934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:42.278132 22448 raft_consensus.cc:1261] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Refusing update from remote peer 4df5b534adfb4e929abae771318a2934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:42.279057 22584 consensus_queue.cc:1035] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:42.279449 22578 consensus_queue.cc:1035] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:42.290649 22590 mvcc.cc:205] Tried to move back new op lower bound from 7317053203562004480 to 7317053203264253952. Current Snapshot: MvccSnapshot[applied={T|T < 7317053203562004480}]
I20260810 19:25:42.309716 21714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:42.309994 21714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:42.313962 21714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Sending DeleteTablet for 3 replicas of tablet 66f1949d37af44a9abe5155f2d42636c
I20260810 19:25:42.315246 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f1949d37af44a9abe5155f2d42636c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:42 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:35858
I20260810 19:25:42.315312 22159 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f1949d37af44a9abe5155f2d42636c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:42 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:42650
I20260810 19:25:42.315416 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f1949d37af44a9abe5155f2d42636c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:42 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:44656
I20260810 19:25:42.317263 21712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:42.318011 21712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:42.326512 21932 tablet_service.cc:1467] Processing CreateTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 (DEFAULT_TABLE table=test-table [id=e7878c34210b416f818987814d7ccfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:42.326656 22406 tablet_service.cc:1467] Processing CreateTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 (DEFAULT_TABLE table=test-table [id=e7878c34210b416f818987814d7ccfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:42.327152 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f93fb4ede74cf59a6ca97661f12807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.327167 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f93fb4ede74cf59a6ca97661f12807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.330013 22158 tablet_service.cc:1467] Processing CreateTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 (DEFAULT_TABLE table=test-table [id=e7878c34210b416f818987814d7ccfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:42.330575 22158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f93fb4ede74cf59a6ca97661f12807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.334396 22566 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Bootstrap starting.
I20260810 19:25:42.335991 22566 tablet_bootstrap.cc:654] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.336985 22563 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Bootstrap starting.
I20260810 19:25:42.338796 22563 tablet_bootstrap.cc:654] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.346904 22565 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Bootstrap starting.
I20260810 19:25:42.348804 22565 tablet_bootstrap.cc:654] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.350523 22563 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: No bootstrap required, opened a new log
I20260810 19:25:42.350745 22563 ts_tablet_manager.cc:1397] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260810 19:25:42.351655 22563 raft_consensus.cc:348] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.352001 22563 raft_consensus.cc:374] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.352142 22563 raft_consensus.cc:729] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fccd521254b453984779baf1c489481, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.352516 22563 consensus_queue.cc:260] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.355624 22563 ts_tablet_manager.cc:1428] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 19:25:42.358989 22566 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: No bootstrap required, opened a new log
I20260810 19:25:42.359143 22566 ts_tablet_manager.cc:1397] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.003s
I20260810 19:25:42.359793 22566 raft_consensus.cc:348] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.359974 22566 raft_consensus.cc:374] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.360064 22566 raft_consensus.cc:729] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.360306 22566 consensus_queue.cc:260] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.360839 22566 ts_tablet_manager.cc:1428] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 19:25:42.367079 22565 tablet_bootstrap.cc:492] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: No bootstrap required, opened a new log
I20260810 19:25:42.367255 22565 ts_tablet_manager.cc:1397] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20260810 19:25:42.368384 22565 raft_consensus.cc:348] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.368647 22565 raft_consensus.cc:374] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.368748 22565 raft_consensus.cc:729] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7038d62fd1ec45bcbb0181a799d46668, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.369295 22565 consensus_queue.cc:260] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.370033 22565 ts_tablet_manager.cc:1428] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:42.370584 22602 tablet_replica.cc:330] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: stopping tablet replica
I20260810 19:25:42.371114 22602 raft_consensus.cc:2229] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:42.371650 22602 raft_consensus.cc:2258] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:42.372506 22603 tablet_replica.cc:330] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: stopping tablet replica
I20260810 19:25:42.373507 22603 raft_consensus.cc:2229] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:42.375602 22602 ts_tablet_manager.cc:1905] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:42.377334 22603 raft_consensus.cc:2258] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:42.378161 22604 tablet_replica.cc:330] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: stopping tablet replica
I20260810 19:25:42.378576 22604 raft_consensus.cc:2229] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:42.379021 22604 raft_consensus.cc:2258] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:42.379628 22603 ts_tablet_manager.cc:1905] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:42.380944 22604 ts_tablet_manager.cc:1905] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:42.383882 22602 ts_tablet_manager.cc:1918] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:25:42.384044 22602 log.cc:1196] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/66f1949d37af44a9abe5155f2d42636c
I20260810 19:25:42.384604 22602 ts_tablet_manager.cc:1939] T 66f1949d37af44a9abe5155f2d42636c P 8fccd521254b453984779baf1c489481: Deleting consensus metadata
I20260810 19:25:42.386399 21690 catalog_manager.cc:4841] TS 8fccd521254b453984779baf1c489481 (127.19.68.2:44467): tablet 66f1949d37af44a9abe5155f2d42636c (table test-table [id=d40e294540e44346bba49029fc9bee3b]) successfully deleted
I20260810 19:25:42.387337 22603 ts_tablet_manager.cc:1918] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:25:42.387871 22603 log.cc:1196] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/66f1949d37af44a9abe5155f2d42636c
I20260810 19:25:42.388023 22604 ts_tablet_manager.cc:1918] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:25:42.388180 22604 log.cc:1196] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/66f1949d37af44a9abe5155f2d42636c
I20260810 19:25:42.388512 22603 ts_tablet_manager.cc:1939] T 66f1949d37af44a9abe5155f2d42636c P 4df5b534adfb4e929abae771318a2934: Deleting consensus metadata
I20260810 19:25:42.388698 22604 ts_tablet_manager.cc:1939] T 66f1949d37af44a9abe5155f2d42636c P 7038d62fd1ec45bcbb0181a799d46668: Deleting consensus metadata
I20260810 19:25:42.389935 21692 catalog_manager.cc:4841] TS 4df5b534adfb4e929abae771318a2934 (127.19.68.1:34159): tablet 66f1949d37af44a9abe5155f2d42636c (table test-table [id=d40e294540e44346bba49029fc9bee3b]) successfully deleted
I20260810 19:25:42.390324 21692 catalog_manager.cc:4841] TS 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347): tablet 66f1949d37af44a9abe5155f2d42636c (table test-table [id=d40e294540e44346bba49029fc9bee3b]) successfully deleted
I20260810 19:25:42.427207 22596 raft_consensus.cc:482] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:42.427506 22596 raft_consensus.cc:504] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.428435 22596 leader_election.cc:288] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.429019 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f93fb4ede74cf59a6ca97661f12807" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481" is_pre_election: true
I20260810 19:25:42.429056 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f93fb4ede74cf59a6ca97661f12807" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668" is_pre_election: true
I20260810 19:25:42.429337 22448 raft_consensus.cc:2454] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.429337 22193 raft_consensus.cc:2454] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.429823 21859 leader_election.cc:302] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 7038d62fd1ec45bcbb0181a799d46668; no voters:
I20260810 19:25:42.430123 22596 raft_consensus.cc:2781] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:42.430260 22596 raft_consensus.cc:482] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:42.430348 22596 raft_consensus.cc:3037] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.432720 22596 raft_consensus.cc:504] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.433320 22596 leader_election.cc:288] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Requested vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.433840 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f93fb4ede74cf59a6ca97661f12807" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668"
I20260810 19:25:42.433840 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f93fb4ede74cf59a6ca97661f12807" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481"
I20260810 19:25:42.434069 22448 raft_consensus.cc:3037] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.434070 22193 raft_consensus.cc:3037] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.437000 22448 raft_consensus.cc:2454] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.437000 22193 raft_consensus.cc:2454] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.437422 21859 leader_election.cc:302] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 7038d62fd1ec45bcbb0181a799d46668; no voters:
I20260810 19:25:42.437753 22596 raft_consensus.cc:2781] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:42.437930 22596 raft_consensus.cc:686] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 1 LEADER]: Becoming Leader. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Running, Role: LEADER
I20260810 19:25:42.438225 22596 consensus_queue.cc:237] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.441123 21712 catalog_manager.cc:5495] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df5b534adfb4e929abae771318a2934 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } health_report { overall_health: UNKNOWN } } }
Aug 10 19:25:42 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:42.502910 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.487537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37673 (local address 127.0.0.1:40480)
0810 19:25:42.488481 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:42.488520 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:42.488679 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:42.489064 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:42.489068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:42.489811 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:42.490034 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.490042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.491523 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.491527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.492479 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.492490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.493492 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.493496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.493636 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.494250 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:42.494268 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:42.496647 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:42.500551 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:42.500562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:42.500567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:42.500976 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:42.501377 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:42.501380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:42.501382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:42.501513 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:42.502081 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:42.502087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:42.502233 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 19:25:42.502400 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 19:25:42.502592 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"spinlock_wait_cycles":896,"thread_start_us":657,"threads_started":1}
I20260810 19:25:42.503189 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.487537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.19.68.62:37673)
0810 19:25:42.488275 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.488282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.488611 (+ 329us) server_negotiation.cc:407] Connection header received
0810 19:25:42.488838 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.488844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.488924 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.489036 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:42.490202 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.491387 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.492612 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.493320 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.493461 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.496874 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:42.496881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:42.496886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:42.496924 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:42.500431 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:42.501104 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:42.501110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:42.501114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:42.501263 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:42.501610 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:42.501615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:42.501619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:42.501966 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:42.502100 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.502515 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.502758 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":397,"threads_started":1}
I20260810 19:25:42.508203 21712 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:40480:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:25:42.509313 21712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:42.515873 21933 tablet_service.cc:1467] Processing CreateTablet for tablet 36a94c29b147489ea2732a935d2dd609 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b6e39f2606449ae91c6ec5247cb489d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:42.516047 22159 tablet_service.cc:1467] Processing CreateTablet for tablet 36a94c29b147489ea2732a935d2dd609 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b6e39f2606449ae91c6ec5247cb489d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:42.516544 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a94c29b147489ea2732a935d2dd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.516378 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 36a94c29b147489ea2732a935d2dd609 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b6e39f2606449ae91c6ec5247cb489d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:42.516578 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a94c29b147489ea2732a935d2dd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.516942 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a94c29b147489ea2732a935d2dd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:42.523587 22566 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: Bootstrap starting.
I20260810 19:25:42.524405 22563 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: Bootstrap starting.
I20260810 19:25:42.525418 22566 tablet_bootstrap.cc:654] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.525916 22563 tablet_bootstrap.cc:654] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.526840 22565 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: Bootstrap starting.
I20260810 19:25:42.527873 22566 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: No bootstrap required, opened a new log
I20260810 19:25:42.528038 22566 ts_tablet_manager.cc:1397] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:25:42.528215 22565 tablet_bootstrap.cc:654] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:42.528746 22566 raft_consensus.cc:348] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.529067 22566 raft_consensus.cc:374] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.529174 22566 raft_consensus.cc:729] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.529448 22566 consensus_queue.cc:260] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.530055 22596 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.530344 22596 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.530455 22565 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: No bootstrap required, opened a new log
I20260810 19:25:42.530602 22565 ts_tablet_manager.cc:1397] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 19:25:42.531277 22565 raft_consensus.cc:348] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.531517 22565 raft_consensus.cc:374] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.531548 22566 ts_tablet_manager.cc:1428] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 19:25:42.531597 22565 raft_consensus.cc:729] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7038d62fd1ec45bcbb0181a799d46668, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.531833 22565 consensus_queue.cc:260] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.532349 22565 ts_tablet_manager.cc:1428] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 19:25:42.532269 22577 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.532617 22577 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.540560 22563 tablet_bootstrap.cc:492] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: No bootstrap required, opened a new log
I20260810 19:25:42.540748 22563 ts_tablet_manager.cc:1397] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260810 19:25:42.541572 22563 raft_consensus.cc:348] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.541831 22563 raft_consensus.cc:374] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:42.541922 22563 raft_consensus.cc:729] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fccd521254b453984779baf1c489481, State: Initialized, Role: FOLLOWER
I20260810 19:25:42.542187 22563 consensus_queue.cc:260] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.542825 22563 ts_tablet_manager.cc:1428] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 19:25:42.542721 22580 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.543118 22580 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.783526 22596 raft_consensus.cc:482] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:42.783758 22596 raft_consensus.cc:504] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.784399 22596 leader_election.cc:288] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.784919 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a94c29b147489ea2732a935d2dd609" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481" is_pre_election: true
I20260810 19:25:42.784924 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a94c29b147489ea2732a935d2dd609" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668" is_pre_election: true
I20260810 19:25:42.785202 22448 raft_consensus.cc:2454] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.785184 22193 raft_consensus.cc:2454] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 0.
I20260810 19:25:42.785643 21859 leader_election.cc:302] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 7038d62fd1ec45bcbb0181a799d46668, 8fccd521254b453984779baf1c489481; no voters:
I20260810 19:25:42.785943 22596 raft_consensus.cc:2781] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:42.786059 22596 raft_consensus.cc:482] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:42.786165 22596 raft_consensus.cc:3037] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.788167 22596 raft_consensus.cc:504] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.788697 22596 leader_election.cc:288] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Requested vote from peers 8fccd521254b453984779baf1c489481 (127.19.68.2:44467), 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347)
I20260810 19:25:42.789165 22448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a94c29b147489ea2732a935d2dd609" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7038d62fd1ec45bcbb0181a799d46668"
I20260810 19:25:42.789193 22193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a94c29b147489ea2732a935d2dd609" candidate_uuid: "4df5b534adfb4e929abae771318a2934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fccd521254b453984779baf1c489481"
I20260810 19:25:42.789377 22448 raft_consensus.cc:3037] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.789391 22193 raft_consensus.cc:3037] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:42.791532 22448 raft_consensus.cc:2454] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.791617 22193 raft_consensus.cc:2454] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df5b534adfb4e929abae771318a2934 in term 1.
I20260810 19:25:42.792019 21858 leader_election.cc:302] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df5b534adfb4e929abae771318a2934, 8fccd521254b453984779baf1c489481; no voters:
I20260810 19:25:42.792292 22596 raft_consensus.cc:2781] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:42.792455 22596 raft_consensus.cc:686] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [term 1 LEADER]: Becoming Leader. State: Replica: 4df5b534adfb4e929abae771318a2934, State: Running, Role: LEADER
I20260810 19:25:42.792698 22596 consensus_queue.cc:237] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } }
I20260810 19:25:42.794162 22578 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: TxnStatusTablet state changed. Reason: New leader 4df5b534adfb4e929abae771318a2934. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.794381 22578 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:42.794991 22627 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:25:42.795301 22627 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:25:42.795724 21712 catalog_manager.cc:5495] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df5b534adfb4e929abae771318a2934 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } health_report { overall_health: UNKNOWN } } }
I20260810 19:25:42.842120 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.836603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34159 (local address 127.0.0.1:35866)
0810 19:25:42.836843 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:42.836864 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:42.837006 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:42.837585 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:42.837590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:42.837603 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:42.837795 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.837805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.838953 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.838959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.840098 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.840111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.840894 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.840900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.841056 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.841146 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:42.841554 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:42.841588 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:25:42.841661 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 19:25:42.841823 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:25:42.842156 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.836652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.19.68.1:34159)
0810 19:25:42.837306 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.837310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.837325 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:25:42.837380 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.837384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.837444 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.837534 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:42.837976 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.838817 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.840238 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.840755 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.840881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.841349 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:42.841454 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:42.841530 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.841723 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.841803 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260810 19:25:42.847059 22448 raft_consensus.cc:1261] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Refusing update from remote peer 4df5b534adfb4e929abae771318a2934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:42.847121 22193 raft_consensus.cc:1261] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Refusing update from remote peer 4df5b534adfb4e929abae771318a2934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:42.847640 22578 consensus_queue.cc:1035] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:42.847908 22596 consensus_queue.cc:1035] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:42.859395 22577 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: TxnStatusTablet state changed. Reason: New leader 4df5b534adfb4e929abae771318a2934. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.859706 22577 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.860178 22580 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: TxnStatusTablet state changed. Reason: New leader 4df5b534adfb4e929abae771318a2934. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.860430 22580 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.873454 22578 consensus_queue.cc:1035] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 19:25:42.878638 22634 mvcc.cc:205] Tried to move back new op lower bound from 7317053205895774208 to 7317053205681418240. Current Snapshot: MvccSnapshot[applied={T|T < 7317053205895774208}]
I20260810 19:25:42.885234 22650 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.885643 22650 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:42.894632 22580 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.894939 22580 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 8fccd521254b453984779baf1c489481: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.895766 22650 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.896062 22650 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 4df5b534adfb4e929abae771318a2934: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:42.896876 22577 tablet_replica.cc:439] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4df5b534adfb4e929abae771318a2934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df5b534adfb4e929abae771318a2934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34159 } } peers { permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 } } peers { permanent_uuid: "7038d62fd1ec45bcbb0181a799d46668" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34347 } } }
I20260810 19:25:42.897190 22577 tablet_replica.cc:442] T 36a94c29b147489ea2732a935d2dd609 P 7038d62fd1ec45bcbb0181a799d46668: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:42.903621 22650 consensus_queue.cc:1035] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fccd521254b453984779baf1c489481" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:42.936446 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.928572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:49669 (local address 127.19.68.1:34159)
0810 19:25:42.928790 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:42.928796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:42.930012 (+ 1216us) server_negotiation.cc:407] Connection header received
0810 19:25:42.930513 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:42.930519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:42.930634 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:42.930760 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:42.931687 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.932532 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.934699 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.935132 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.935228 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.935789 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:42.935886 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:42.935956 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:42.936102 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:25:42.936177 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":89}
I20260810 19:25:42.936771 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:42.918817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34159 (local address 127.19.68.1:49669)
0810 19:25:42.929915 (+ 11098us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:42.929935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:42.930316 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:42.930776 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:42.930784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:42.930797 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:42.931320 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.931333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.932675 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.932684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.934542 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:42.934558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:42.935321 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:42.935329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:42.935527 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:42.935603 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:42.935982 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:42.935994 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 19:25:42.936084 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:25:42.936239 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10844,"mutex_wait_us":41,"thread_start_us":11189,"threads_started":1}
I20260810 19:25:43.006429 21712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:43.006659 21712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:43.009624 21712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0159c11056b24df48afeda76edb6f237: Sending DeleteTablet for 3 replicas of tablet f4f93fb4ede74cf59a6ca97661f12807
I20260810 19:25:43.010351 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:43 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:35858
I20260810 19:25:43.010478 22159 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:43 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:42650
I20260810 19:25:43.010532 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f93fb4ede74cf59a6ca97661f12807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:43 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:44656
I20260810 19:25:43.021157 22671 tablet_replica.cc:330] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: stopping tablet replica
I20260810 19:25:43.022189 22671 raft_consensus.cc:2229] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:43.022688 22671 raft_consensus.cc:2258] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:43.023574 22667 tablet_replica.cc:330] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: stopping tablet replica
I20260810 19:25:43.023790 22667 raft_consensus.cc:2229] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:43.024271 22667 raft_consensus.cc:2258] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:43.024371 22671 ts_tablet_manager.cc:1905] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:43.026604 22667 ts_tablet_manager.cc:1905] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:43.030879 22670 tablet_replica.cc:330] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: stopping tablet replica
I20260810 19:25:43.031091 22670 raft_consensus.cc:2229] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:43.031373 22670 raft_consensus.cc:2258] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:43.033071 22671 ts_tablet_manager.cc:1918] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:43.033284 22671 log.cc:1196] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/f4f93fb4ede74cf59a6ca97661f12807
I20260810 19:25:43.033793 22671 ts_tablet_manager.cc:1939] T f4f93fb4ede74cf59a6ca97661f12807 P 7038d62fd1ec45bcbb0181a799d46668: Deleting consensus metadata
I20260810 19:25:43.035030 21692 catalog_manager.cc:4841] TS 7038d62fd1ec45bcbb0181a799d46668 (127.19.68.3:34347): tablet f4f93fb4ede74cf59a6ca97661f12807 (table test-table [id=e7878c34210b416f818987814d7ccfd0]) successfully deleted
I20260810 19:25:43.037211 22670 ts_tablet_manager.cc:1905] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:43.042477 22667 ts_tablet_manager.cc:1918] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:43.042692 22667 log.cc:1196] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/f4f93fb4ede74cf59a6ca97661f12807
I20260810 19:25:43.043529 22667 ts_tablet_manager.cc:1939] T f4f93fb4ede74cf59a6ca97661f12807 P 4df5b534adfb4e929abae771318a2934: Deleting consensus metadata
I20260810 19:25:43.043555 22670 ts_tablet_manager.cc:1918] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:43.043696 22670 log.cc:1196] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/f4f93fb4ede74cf59a6ca97661f12807
I20260810 19:25:43.044142 22670 ts_tablet_manager.cc:1939] T f4f93fb4ede74cf59a6ca97661f12807 P 8fccd521254b453984779baf1c489481: Deleting consensus metadata
I20260810 19:25:43.044863 21692 catalog_manager.cc:4841] TS 4df5b534adfb4e929abae771318a2934 (127.19.68.1:34159): tablet f4f93fb4ede74cf59a6ca97661f12807 (table test-table [id=e7878c34210b416f818987814d7ccfd0]) successfully deleted
I20260810 19:25:43.045270 21690 catalog_manager.cc:4841] TS 8fccd521254b453984779baf1c489481 (127.19.68.2:44467): tablet f4f93fb4ede74cf59a6ca97661f12807 (table test-table [id=e7878c34210b416f818987814d7ccfd0]) successfully deleted
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:43.053995 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:43.023575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.19.68.1:34159)
0810 19:25:43.023778 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:43.023784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:43.035879 (+ 12095us) server_negotiation.cc:407] Connection header received
0810 19:25:43.036067 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:43.036073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:43.036166 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:43.036275 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:43.038218 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.039447 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.041430 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.042067 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.042174 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:43.047148 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:43.047166 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:43.047185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:43.047226 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:43.051132 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.051842 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.051850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.051852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.052001 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.052378 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.052382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.052388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.052801 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:43.053169 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:43.053466 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 19:25:43.053681 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 19:25:43.054726 22044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35882
I20260810 19:25:43.071594 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:43.057555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.19.68.62:37673)
0810 19:25:43.058131 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:43.058135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:43.058152 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:25:43.058199 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:43.058203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:43.058280 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:43.058367 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:43.060032 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.060899 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.062656 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.063094 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.063195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:43.065939 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:43.065944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:43.065947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:43.065996 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:43.068581 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.069486 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.069491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.069493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.069636 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.070051 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.070054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.070057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.070456 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:43.070643 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:43.071054 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 19:25:43.071234 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":311,"threads_started":1}
W20260810 19:25:43.072213 21712 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40492
I20260810 19:25:43.074137 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21832
I20260810 19:25:43.088961 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:43.075178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.19.68.1:34159)
0810 19:25:43.075379 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:43.075385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:43.076022 (+ 637us) server_negotiation.cc:407] Connection header received
0810 19:25:43.076137 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:43.076143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:43.076228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:43.076330 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:43.077479 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.078698 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.079930 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.080754 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.080896 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:43.083158 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:43.083164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:43.083169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:43.083209 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:43.086407 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.087000 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.087005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.087009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.087179 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.087558 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.087563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.087567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.087900 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:43.088191 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:43.088332 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 19:25:43.088574 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 19:25:43.089593 22044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35896
W20260810 19:25:43.090190 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21832: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:43.090381 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21832
I20260810 19:25:43.125560 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22057
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:25:43.140149 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:43.126379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.19.68.2:44467)
0810 19:25:43.126873 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:43.126877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:43.126894 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:25:43.126955 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:43.126961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:43.127048 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:43.127142 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:43.128400 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.129198 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.130479 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.130924 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.131029 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:43.134852 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:43.134867 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:43.134881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:43.134913 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:43.137513 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.138136 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.138143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.138146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.138304 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.138652 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.138656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.138658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.138990 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:43.139310 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:43.139560 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 19:25:43.139811 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
W20260810 19:25:43.140758 22275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42656
W20260810 19:25:43.141324 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:43.141470 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22057
I20260810 19:25:43.165544 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22294
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:25:43.180368 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:43.166286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.19.68.3:34347)
0810 19:25:43.166729 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:43.166733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:43.166751 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:25:43.166807 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:43.166810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:43.166872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:43.166971 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:43.168193 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.168910 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.170195 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:43.170657 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:43.170758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:43.174766 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:43.174782 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:43.174798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:43.174833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:43.177683 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.178464 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.178477 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.178480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.178618 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:43.178957 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:43.178960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:43.178962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:43.179298 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:43.179541 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:43.179823 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 19:25:43.180005 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
W20260810 19:25:43.181149 22524 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666
W20260810 19:25:43.181790 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22294: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:43.181982 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22294
I20260810 19:25:43.207090 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21663
I20260810 19:25:43.208348 21776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:43.323230 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21663
2026-08-10T19:25:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4998 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[22693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[22693](info): set up 2 sockets
Aug 10 19:25:43 dist-test-slave-2j7r krb5kdc[22693](info): commencing operation
krb5kdc: starting...
W20260810 19:25:44.556770 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:44 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:44.774739 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:45347
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:45.095361 22714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:45.095647 22714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:45.095774 22714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:45.102979 22714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:45.103073 22714 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:25:45.103127 22714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:45.103214 22714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:45.103278 22714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:45.103364 22714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:45.113112 22714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:45347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:45.114410 22714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:45.115720 22714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:45.127386 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:45.127437 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:45.127455 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:45.128343 22714 server_base.cc:1034] running on GCE node
I20260810 19:25:45.129172 22714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:45.130605 22714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:45.131820 22714 hybrid_clock.cc:648] HybridClock initialized: now 1786389945131749 us; error 44 us; skew 500 ppm
I20260810 19:25:45.132120 22714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:45 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:45.135200 22714 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:45.136703 22714 webserver.cc:460] Webserver started at http://127.19.68.62:38005/ using document root <none> and password file <none>
I20260810 19:25:45.137660 22714 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:45.137753 22714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:45.138019 22714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:45.140879 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "77abdc4eb7734bf6a3da9ccd55b613ff"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "76fc56d2f9a293e8fd34cbb8157b3791"
server_key_iv: "7c32bb5832f0eee7275073c0bc8c9f23"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.141638 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "77abdc4eb7734bf6a3da9ccd55b613ff"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "76fc56d2f9a293e8fd34cbb8157b3791"
server_key_iv: "7c32bb5832f0eee7275073c0bc8c9f23"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.146656 22714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:25:45.150027 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:45.151413 22714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:45.151556 22714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "77abdc4eb7734bf6a3da9ccd55b613ff"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "76fc56d2f9a293e8fd34cbb8157b3791"
server_key_iv: "7c32bb5832f0eee7275073c0bc8c9f23"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.151736 22714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:45.163450 22714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:45.167179 22714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:45.167411 22714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:45.187255 22714 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:45347
I20260810 19:25:45.187276 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:45347 every 8 connection(s)
I20260810 19:25:45.188516 22714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:45.192333 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:45.194728 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22714
I20260810 19:25:45.194952 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:45.195735 19728 external_mini_cluster.cc:1428] Setting key 5cd67cf8d388b9c2d71ee1923f511dbb
I20260810 19:25:45.204561 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Bootstrap starting.
I20260810 19:25:45.208351 22850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:45.209689 22850 log.cc:824] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:25:45 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:45.212651 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: No bootstrap required, opened a new log
I20260810 19:25:45.218451 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:45.197826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.19.68.62:45347)
0810 19:25:45.198626 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:45.198638 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:45.198679 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:25:45.200324 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:45.200349 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:45.200715 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:45.201065 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:45.202825 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.204073 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.205789 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.206484 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.206621 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:45.211467 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:45.211475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:45.211490 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:45.211531 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:45.214493 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.215154 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.215161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.215170 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.215347 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.215714 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.215717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.215719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.216256 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:45.216574 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:45.217009 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 19:25:45.217449 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":392,"threads_started":1}
I20260810 19:25:45.220382 22850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } }
I20260810 19:25:45.220733 22850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:45.220844 22850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77abdc4eb7734bf6a3da9ccd55b613ff, State: Initialized, Role: FOLLOWER
I20260810 19:25:45.221453 22850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } }
I20260810 19:25:45.221666 22850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:45.221760 22850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:45.221873 22850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:45.223919 22850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } }
I20260810 19:25:45.224370 22850 leader_election.cc:302] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77abdc4eb7734bf6a3da9ccd55b613ff; no voters:
I20260810 19:25:45.224993 22850 leader_election.cc:288] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:45.225190 22864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:45.226112 22864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [term 1 LEADER]: Becoming Leader. State: Replica: 77abdc4eb7734bf6a3da9ccd55b613ff, State: Running, Role: LEADER
I20260810 19:25:45.226598 22864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } }
I20260810 19:25:45.227124 22850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:45.230283 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } } }
I20260810 19:25:45.230688 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:45.231016 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77abdc4eb7734bf6a3da9ccd55b613ff. Latest consensus state: current_term: 1 leader_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77abdc4eb7734bf6a3da9ccd55b613ff" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45347 } } }
I20260810 19:25:45.231264 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:45.233685 22879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:45.238734 22879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:45.247808 22879 catalog_manager.cc:1347] Generated new cluster ID: 274304222c014310ab43a119f5615d64
I20260810 19:25:45.247925 22879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:45.262280 22879 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:45.263769 22879 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:45.274571 22879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Generated new TSK 0
I20260810 19:25:45.275359 22879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:45.331979 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:45347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:45.676120 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:45.676421 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:45.676577 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:45.684079 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:45.684190 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:45.684342 22903 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:25:45.684417 22903 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:45.684535 22903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:25:45.694278 22903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:45.695559 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:45.696895 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:45.710166 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:45.710341 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:45.710673 22928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:45.711011 22903 server_base.cc:1034] running on GCE node
I20260810 19:25:45.711616 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:45.712545 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:45.713744 22903 hybrid_clock.cc:648] HybridClock initialized: now 1786389945713692 us; error 47 us; skew 500 ppm
I20260810 19:25:45.714039 22903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:45 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:45.717044 22903 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:25:45.718611 22903 webserver.cc:460] Webserver started at http://127.19.68.1:38637/ using document root <none> and password file <none>
I20260810 19:25:45.719280 22903 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:45.719399 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:45.719692 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:45.722579 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "182555744a084d64b8a48065c49fca6e"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "f06e015d013c10d5898db31561ace728"
server_key_iv: "b27a5ca6ee60c89109dddbc68723daa5"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.723308 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "182555744a084d64b8a48065c49fca6e"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "f06e015d013c10d5898db31561ace728"
server_key_iv: "b27a5ca6ee60c89109dddbc68723daa5"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.728223 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 19:25:45.731719 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:45.732936 22903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:25:45.733132 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "182555744a084d64b8a48065c49fca6e"
format_stamp: "Formatted at 2026-08-10 19:25:45 on dist-test-slave-2j7r"
server_key: "f06e015d013c10d5898db31561ace728"
server_key_iv: "b27a5ca6ee60c89109dddbc68723daa5"
server_key_version: "encryptionkey@0"
I20260810 19:25:45.733330 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:45.748961 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:45.752854 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:45.753139 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:45.755623 22903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:45.755748 22903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:45.755861 22903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:45.755950 22903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:25:45 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:45.777191 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:45.756280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:54419 (local address 127.19.68.62:45347)
0810 19:25:45.756757 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:45.756764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:45.757907 (+ 1143us) server_negotiation.cc:407] Connection header received
0810 19:25:45.759924 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:45.759930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:45.760015 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:45.760109 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:45.762105 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.763246 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.765679 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.766096 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.766209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:45.770907 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:45.770912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:45.770915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:45.770948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:45.773872 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.774623 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.774629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.774632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.774813 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.775272 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.775278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.775282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.775690 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:45.775863 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:45.776446 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 19:25:45.776720 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260810 19:25:45.778194 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:45.756955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.1:54419)
0810 19:25:45.757753 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:45.757788 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:45.759611 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:45.760342 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:45.760349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:45.760966 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:45.761848 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:45.761870 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.763406 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.763412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:45.765487 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:45.765505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.766421 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.766431 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:45.766728 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:45.767435 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:45.767472 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:45.770666 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:45.774010 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:45.774019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:45.774033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:45.774495 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:45.774954 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:45.774959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:45.774963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:45.775142 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:45.775813 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:45.775818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:45.776116 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 19:25:45.776325 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 19:25:45.776681 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260810 19:25:45.797261 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:45.784122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:60669 (local address 127.19.68.62:45347)
0810 19:25:45.784259 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:45.784263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:45.784875 (+ 612us) server_negotiation.cc:407] Connection header received
0810 19:25:45.785033 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:45.785038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:45.785122 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:45.785208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:45.786589 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.787495 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.789416 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.789921 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.790031 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:45.791840 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:45.791844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:45.791848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:45.791878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:45.794241 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.795130 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.795135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.795139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.795314 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:45.795708 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:45.795714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:45.795718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:45.796109 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:45.796268 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:45.796671 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 19:25:45.796893 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:25:45.797395 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:45.784188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.1:60669)
0810 19:25:45.784764 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:45.784780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:45.784913 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:45.785249 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:45.785255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:45.785929 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:45.786379 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:45.786397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.787627 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.787636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:45.789234 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:45.789252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:45.790119 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:45.790126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:45.790281 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:45.790940 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:45.790957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:45.791667 (+ 710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:45.794406 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:45.794412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:45.794416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:45.795008 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:45.795454 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:45.795460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:45.795464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:45.795598 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:45.796230 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:45.796235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:45.796428 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 19:25:45.796548 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:25:45.796855 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":312,"threads_started":1}
I20260810 19:25:45.805780 22903 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34121
I20260810 19:25:45.805820 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34121 every 8 connection(s)
I20260810 19:25:45.806918 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:25:45.808683 23147 heartbeater.cc:347] Connected to a master server at 127.19.68.62:45347
I20260810 19:25:45.808884 23147 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:45.809562 23147 heartbeater.cc:511] Master 127.19.68.62:45347 requested a full tablet report, sending...
I20260810 19:25:45.811028 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22903
I20260810 19:25:45.811211 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:25:45.811580 22766 ts_manager.cc:194] Registered new tserver with Master: 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121)
I20260810 19:25:45.811847 19728 external_mini_cluster.cc:1428] Setting key da442b772b163affa3a7993f4b86cd02
I20260810 19:25:45.813297 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:54419
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:25:45.855916 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:45347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:46.178153 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:46.178424 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:46.178560 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:46.185729 23155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:46.185825 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:46.185947 23155 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:25:46.186030 23155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:46.186133 23155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:25:46.195513 23155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:46.196717 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:46.198097 23155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:46.210983 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:46.210920 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:46.210961 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:46.211601 23155 server_base.cc:1034] running on GCE node
I20260810 19:25:46.212214 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:46.213184 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:46.214381 23155 hybrid_clock.cc:648] HybridClock initialized: now 1786389946214315 us; error 56 us; skew 500 ppm
I20260810 19:25:46.214716 23155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:46.217929 23155 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:25:46.219483 23155 webserver.cc:460] Webserver started at http://127.19.68.2:36523/ using document root <none> and password file <none>
I20260810 19:25:46.220134 23155 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:46.220242 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:46.220559 23155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:46.223562 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "1524942834d844a89ce0f5f6986a1dc5"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "d03dd4351deccbbb9f148bd040037877"
server_key_iv: "79e6cb6d050b994624ec7cac921d7d6f"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.224316 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "1524942834d844a89ce0f5f6986a1dc5"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "d03dd4351deccbbb9f148bd040037877"
server_key_iv: "79e6cb6d050b994624ec7cac921d7d6f"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.229786 23155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 19:25:46.233312 23178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:46.234714 23155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:46.234885 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "1524942834d844a89ce0f5f6986a1dc5"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "d03dd4351deccbbb9f148bd040037877"
server_key_iv: "79e6cb6d050b994624ec7cac921d7d6f"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.235078 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:46.248761 23155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:46.252408 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:46.252681 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:46.255396 23155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:46.255504 23155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:46.255627 23155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:46.255716 23155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:46.279958 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.255834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:39243 (local address 127.19.68.62:45347)
0810 19:25:46.256031 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.256036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.257451 (+ 1415us) server_negotiation.cc:407] Connection header received
0810 19:25:46.259443 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.259450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.259583 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.259711 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:46.261822 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.262829 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.265829 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.266475 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.266633 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.272117 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:46.272124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:46.272129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:46.272175 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:46.275799 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.276890 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.276896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.276900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.277060 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.277636 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.277642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.277646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.278048 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:46.278213 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.279344 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.279597 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:25:46.281890 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.256392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.2:39243)
0810 19:25:46.257264 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.257316 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.259169 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.260007 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.260018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.260752 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:46.261635 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.261670 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.263018 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.263024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.265614 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.265633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.266780 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.266788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.267114 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.267941 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:46.267974 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:46.271764 (+ 3790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:46.275957 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.275968 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.275988 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.276723 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.277258 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.277263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.277267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.277515 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.278212 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:46.278221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:46.278766 (+ 545us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.279180 (+ 414us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.279822 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":373,"threads_started":1}
I20260810 19:25:46.293460 23155 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:40857
I20260810 19:25:46.293533 23366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:40857 every 8 connection(s)
I20260810 19:25:46.294939 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:25:46.297389 23367 heartbeater.cc:347] Connected to a master server at 127.19.68.62:45347
I20260810 19:25:46.297695 23367 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:46.298424 23367 heartbeater.cc:511] Master 127.19.68.62:45347 requested a full tablet report, sending...
I20260810 19:25:46.299887 22766 ts_manager.cc:194] Registered new tserver with Master: 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857)
I20260810 19:25:46.300788 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:39243
I20260810 19:25:46.304107 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.289795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.2:56403)
0810 19:25:46.291251 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.291285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.291450 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.291799 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.291803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.292392 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:46.292862 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.292878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.294170 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.294176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.295693 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.295711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.296871 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.296875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.297014 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.297663 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:46.297683 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:46.298456 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:46.301783 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.301788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.301791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.302201 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.302583 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.302585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.302588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.302697 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.303353 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:46.303356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:46.303494 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.303599 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.303816 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":1093,"threads_started":1}
I20260810 19:25:46.304158 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23155
I20260810 19:25:46.304224 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.290334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:56403 (local address 127.19.68.62:45347)
0810 19:25:46.290516 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.290522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.291371 (+ 849us) server_negotiation.cc:407] Connection header received
0810 19:25:46.291584 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.291589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.291666 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.291747 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:46.293033 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.294036 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.295888 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.296400 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.296473 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.298696 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:46.298702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:46.298707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:46.298744 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:46.301641 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.302318 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.302323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.302326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.302488 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.302789 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.302794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.302797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.303223 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:46.303371 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.303690 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.303928 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:25:46.304347 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:25:46.304857 19728 external_mini_cluster.cc:1428] Setting key fa17fe1f37c6e191b53ea1fa6a29525d
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:25:46.349670 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:45347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:46.670411 23372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:46.670698 23372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:46.670845 23372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:46.678279 23372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:46.678411 23372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:46.678535 23372 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:25:46.678610 23372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:25:46.678719 23372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:25:46.688405 23372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:46.689704 23372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:46.690971 23372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:46.703444 23381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:46.703392 23383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:46.703385 23388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:46.704229 23372 server_base.cc:1034] running on GCE node
I20260810 19:25:46.704780 23372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:46.705734 23372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:46.706980 23372 hybrid_clock.cc:648] HybridClock initialized: now 1786389946706936 us; error 39 us; skew 500 ppm
I20260810 19:25:46.707334 23372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:46.710640 23372 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:25:46.712062 23372 webserver.cc:460] Webserver started at http://127.19.68.3:36013/ using document root <none> and password file <none>
I20260810 19:25:46.712708 23372 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:46.712821 23372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:46.713155 23372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:46.716003 23372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "b0bd8cc4b772d752e95918cbaec29a99"
server_key_iv: "99d7309499d16e40a1f9007c32402879"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.716722 23372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "b0bd8cc4b772d752e95918cbaec29a99"
server_key_iv: "99d7309499d16e40a1f9007c32402879"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.721725 23372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:25:46.725270 23399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:46.726608 23372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:25:46.726738 23372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
format_stamp: "Formatted at 2026-08-10 19:25:46 on dist-test-slave-2j7r"
server_key: "b0bd8cc4b772d752e95918cbaec29a99"
server_key_iv: "99d7309499d16e40a1f9007c32402879"
server_key_version: "encryptionkey@0"
I20260810 19:25:46.726900 23372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:46.744153 23372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:46.752430 23372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:46.752710 23372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:46.755499 23372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:46.755620 23372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:46.755755 23372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:46.755844 23372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:46.777273 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.755900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:48323 (local address 127.19.68.62:45347)
0810 19:25:46.756068 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.756074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.757625 (+ 1551us) server_negotiation.cc:407] Connection header received
0810 19:25:46.759737 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.759743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.759823 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.759921 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:46.762060 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.763222 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.765538 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.766191 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.766336 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.770577 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:46.770582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:46.770585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:46.770620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:46.773833 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.774494 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.774499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.774504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.774678 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.775076 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.775081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.775084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.775570 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:46.775776 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.776482 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.776771 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 19:25:46.777904 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.756531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.3:48323)
0810 19:25:46.757452 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.757503 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.759420 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.760207 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.760219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.761004 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:46.761832 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.761862 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.763404 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.763410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.765381 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.765394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.766443 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.766454 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.766697 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.767381 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:46.767403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:46.770357 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:46.773962 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.773969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.773979 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.774380 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.774778 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.774781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.774784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.774968 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.775724 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:46.775729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:46.776048 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.776262 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.776630 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":410,"threads_started":1}
I20260810 19:25:46.798156 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.783822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.19.68.3:40573)
0810 19:25:46.784533 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.784558 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.784710 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.785039 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.785045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.785717 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:46.786174 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.786190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.787150 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.787156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.788997 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.789017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.790494 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.790503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.790753 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.791435 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:46.791452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:46.792549 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:46.795523 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.795528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.795532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.795922 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.796375 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:46.796378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:46.796384 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:46.796504 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:46.797036 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:46.797041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:46.797265 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.797432 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.797779 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260810 19:25:46.798440 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.783908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:40573 (local address 127.19.68.62:45347)
0810 19:25:46.784108 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.784114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.784646 (+ 532us) server_negotiation.cc:407] Connection header received
0810 19:25:46.784831 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.784834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.784922 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.784994 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:46.786345 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.787021 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.789273 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.790016 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.790124 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.792727 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:46.792732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:46.792735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:46.792764 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:46.795398 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.796085 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.796090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.796093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.796248 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.796611 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.796616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.796620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.796924 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:46.797054 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.797836 (+ 782us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.798078 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:25:46.804523 23372 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:35855
I20260810 19:25:46.804522 23623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:35855 every 8 connection(s)
I20260810 19:25:46.805842 23372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:25:46.808566 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23372
I20260810 19:25:46.808727 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:25:46.809238 19728 external_mini_cluster.cc:1428] Setting key 9a97a6ee9d58fd78c37332e184e8b0b3
I20260810 19:25:46.807870 23624 heartbeater.cc:347] Connected to a master server at 127.19.68.62:45347
I20260810 19:25:46.811113 23624 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:46.811738 23624 heartbeater.cc:511] Master 127.19.68.62:45347 requested a full tablet report, sending...
I20260810 19:25:46.813011 22766 ts_manager.cc:194] Registered new tserver with Master: e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:46.813858 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:48323
I20260810 19:25:46.815881 23147 heartbeater.cc:503] Master 127.19.68.62:45347 was elected leader, sending a full tablet report...
I20260810 19:25:46.823876 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:25:46 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:46.878278 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.861156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.19.68.62:45347)
0810 19:25:46.861400 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.861407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.861679 (+ 272us) server_negotiation.cc:407] Connection header received
0810 19:25:46.861866 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.861870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.861963 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.862059 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:46.863757 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.864870 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.866706 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.867351 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.867444 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.872386 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:46.872392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:46.872395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:46.872441 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:46.875659 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.876274 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.876280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.876283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.876413 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:46.876779 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:46.876784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:46.876788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:46.877099 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:46.877345 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.877661 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.877923 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:25:46.882506 22766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:46.889515 22766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:46.912467 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.902243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.19.68.2:40857)
0810 19:25:46.903568 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.903575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.904334 (+ 759us) server_negotiation.cc:407] Connection header received
0810 19:25:46.905313 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.905321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.905546 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.905724 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:46.906876 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.907790 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.910139 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.910991 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.911113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.911325 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.911941 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.912116 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":1076,"threads_started":1}
I20260810 19:25:46.913321 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.901343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:40857 (local address 127.0.0.1:49584)
0810 19:25:46.904218 (+ 2875us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.904235 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.904366 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.905732 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.905745 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.905770 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:46.906720 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.906731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.907927 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.907932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.909987 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.909999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.911468 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.911475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.911667 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.911716 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.911854 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.912035 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2781,"mutex_wait_us":1,"thread_start_us":3062,"threads_started":1}
I20260810 19:25:46.913666 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.901762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34121 (local address 127.0.0.1:55700)
0810 19:25:46.904626 (+ 2864us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.904650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.904767 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.905517 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.905524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.905566 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:46.906366 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.906380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.908299 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.908303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.910926 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.910941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.912784 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.912793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.912927 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.912963 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.913046 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.913198 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2760,"mutex_wait_us":226,"spinlock_wait_cycles":2432,"thread_start_us":2251,"threads_started":1}
I20260810 19:25:46.914130 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.902337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.19.68.1:34121)
0810 19:25:46.903886 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.903891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.904816 (+ 925us) server_negotiation.cc:407] Connection header received
0810 19:25:46.904911 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.904917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.905134 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.905320 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:46.906564 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.908121 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.911110 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.912140 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.912256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.912462 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.913416 (+ 954us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.913866 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":1327,"threads_started":1}
I20260810 19:25:46.914386 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.901703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.19.68.3:35855)
0810 19:25:46.904894 (+ 3191us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:46.904902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:46.904932 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:25:46.905027 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:46.905034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:46.905296 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:46.906092 (+ 796us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:46.906553 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.908148 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.911103 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.912512 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.912751 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.912974 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:46.913927 (+ 953us) server_negotiation.cc:299] Negotiation successful
0810 19:25:46.914083 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3010,"thread_start_us":2855,"threads_started":1}
I20260810 19:25:46.914536 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:46.901251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:35855 (local address 127.0.0.1:49234)
0810 19:25:46.902912 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:46.902955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:46.903112 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:46.905520 (+ 2408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:46.905527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:46.905585 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:46.906336 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.906363 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.908298 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.908303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.910926 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:46.910941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:46.913134 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:46.913140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:46.913335 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:46.913368 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:25:46.913876 (+ 508us) client_negotiation.cc:240] Negotiation successful
0810 19:25:46.913999 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":120,"spinlock_wait_cycles":23424,"thread_start_us":1271,"threads_started":1}
I20260810 19:25:46.917146 23254 tablet_service.cc:1467] Processing CreateTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (DEFAULT_TABLE table=test-table [id=e7cc86322d474fd7acc684704de3a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:46.917847 23026 tablet_service.cc:1467] Processing CreateTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (DEFAULT_TABLE table=test-table [id=e7cc86322d474fd7acc684704de3a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:46.918869 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd0c34d3d3d42708d7bbadbc2d0b736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:46.918407 23487 tablet_service.cc:1467] Processing CreateTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (DEFAULT_TABLE table=test-table [id=e7cc86322d474fd7acc684704de3a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:46.919235 23026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd0c34d3d3d42708d7bbadbc2d0b736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:46.919974 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd0c34d3d3d42708d7bbadbc2d0b736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:46.932468 23660 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Bootstrap starting.
I20260810 19:25:46.934176 23662 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Bootstrap starting.
I20260810 19:25:46.936286 23660 tablet_bootstrap.cc:654] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:46.937770 23662 tablet_bootstrap.cc:654] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:46.937844 23660 log.cc:824] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:46.937929 23661 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Bootstrap starting.
I20260810 19:25:46.939043 23662 log.cc:824] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:46.941633 23660 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: No bootstrap required, opened a new log
I20260810 19:25:46.941668 23661 tablet_bootstrap.cc:654] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:46.941939 23660 ts_tablet_manager.cc:1397] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 19:25:46.942619 23661 log.cc:824] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:46.945230 23662 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: No bootstrap required, opened a new log
I20260810 19:25:46.945472 23661 tablet_bootstrap.cc:492] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: No bootstrap required, opened a new log
I20260810 19:25:46.945477 23662 ts_tablet_manager.cc:1397] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260810 19:25:46.945744 23661 ts_tablet_manager.cc:1397] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 19:25:46.952117 23660 raft_consensus.cc:348] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.952562 23660 raft_consensus.cc:374] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:46.952661 23660 raft_consensus.cc:729] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Initialized, Role: FOLLOWER
I20260810 19:25:46.952914 23662 raft_consensus.cc:348] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.953317 23662 raft_consensus.cc:374] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:46.953413 23662 raft_consensus.cc:729] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f34e5be0324bd2a5a2bf7155535eec, State: Initialized, Role: FOLLOWER
I20260810 19:25:46.953343 23660 consensus_queue.cc:260] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.953966 23662 consensus_queue.cc:260] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.954591 23367 heartbeater.cc:503] Master 127.19.68.62:45347 was elected leader, sending a full tablet report...
I20260810 19:25:46.955112 23660 ts_tablet_manager.cc:1428] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 19:25:46.955916 23661 raft_consensus.cc:348] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.956338 23661 raft_consensus.cc:374] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:46.956432 23661 raft_consensus.cc:729] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Initialized, Role: FOLLOWER
I20260810 19:25:46.956667 23624 heartbeater.cc:503] Master 127.19.68.62:45347 was elected leader, sending a full tablet report...
I20260810 19:25:46.957353 23662 ts_tablet_manager.cc:1428] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 19:25:46.958105 23661 consensus_queue.cc:260] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:46.959592 23661 ts_tablet_manager.cc:1428] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260810 19:25:47.046691 23671 raft_consensus.cc:482] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 19:25:47.047129 23369 tablet.cc:2369] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:47.047047 23671 raft_consensus.cc:504] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:47.048903 23671 leader_election.cc:288] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855), 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121)
I20260810 19:25:47.056532 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.049298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:47841 (local address 127.19.68.3:35855)
0810 19:25:47.049514 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.049520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.050232 (+ 712us) server_negotiation.cc:407] Connection header received
0810 19:25:47.050447 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.050452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.050513 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.050596 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:47.051179 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.052231 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.054655 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.055506 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.055629 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.055734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.056121 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.056263 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:47.056555 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.049424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:53643 (local address 127.19.68.1:34121)
0810 19:25:47.049643 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.049650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.050576 (+ 926us) server_negotiation.cc:407] Connection header received
0810 19:25:47.050739 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.050745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.050818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.050894 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:47.051497 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.052476 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.054571 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.055449 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.055551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.055632 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.056185 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.056331 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:25:47.056973 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.049156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:35855 (local address 127.19.68.2:47841)
0810 19:25:47.050118 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.050135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.050300 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.050637 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.050642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.050665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:47.051031 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.051043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.052365 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.052370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.054401 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.054416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.055728 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.055736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.055910 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.055953 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.056057 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.056187 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":179,"thread_start_us":632,"threads_started":1}
I20260810 19:25:47.056975 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.049597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34121 (local address 127.19.68.2:53643)
0810 19:25:47.050467 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.050482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.050605 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.050945 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.050950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.050977 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:47.051327 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.051337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.052610 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.052616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.054385 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.054399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.055801 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.055806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.055968 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.055999 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.056130 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.056261 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":50,"thread_start_us":536,"threads_started":1}
W20260810 19:25:47.057500 23626 tablet.cc:2369] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:47.058190 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd0c34d3d3d42708d7bbadbc2d0b736" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" is_pre_election: true
I20260810 19:25:47.058621 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd0c34d3d3d42708d7bbadbc2d0b736" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182555744a084d64b8a48065c49fca6e" is_pre_election: true
I20260810 19:25:47.058859 23529 raft_consensus.cc:2454] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 0.
W20260810 19:25:47.059123 23148 tablet.cc:2369] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:47.059249 23063 raft_consensus.cc:2454] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 0.
I20260810 19:25:47.059942 23180 leader_election.cc:302] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, 182555744a084d64b8a48065c49fca6e; no voters:
I20260810 19:25:47.060321 23671 raft_consensus.cc:2781] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:47.060452 23671 raft_consensus.cc:482] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:47.060554 23671 raft_consensus.cc:3037] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.062770 23671 raft_consensus.cc:504] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:47.063409 23671 leader_election.cc:288] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 election: Requested vote from peers e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855), 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121)
I20260810 19:25:47.063877 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd0c34d3d3d42708d7bbadbc2d0b736" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
I20260810 19:25:47.063956 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd0c34d3d3d42708d7bbadbc2d0b736" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182555744a084d64b8a48065c49fca6e"
I20260810 19:25:47.064100 23529 raft_consensus.cc:3037] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.064152 23063 raft_consensus.cc:3037] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.067293 23063 raft_consensus.cc:2454] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 1.
I20260810 19:25:47.067293 23529 raft_consensus.cc:2454] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 1.
I20260810 19:25:47.067714 23180 leader_election.cc:302] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, 182555744a084d64b8a48065c49fca6e; no voters:
I20260810 19:25:47.068028 23671 raft_consensus.cc:2781] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:47.068483 23671 raft_consensus.cc:686] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Running, Role: LEADER
I20260810 19:25:47.068951 23671 consensus_queue.cc:237] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } }
I20260810 19:25:47.073923 22763 catalog_manager.cc:5495] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "1524942834d844a89ce0f5f6986a1dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } health_report { overall_health: UNKNOWN } } }
Aug 10 19:25:47 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:47.129817 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.115621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:45347 (local address 127.0.0.1:57052)
0810 19:25:47.116364 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.116383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.116551 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.116913 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.116918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.117768 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:47.118071 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.118083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.119153 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.119160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.120187 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.120215 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.121019 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.121026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.121222 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.121895 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:47.121916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:47.124311 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:47.127311 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:47.127325 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:47.127334 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:47.127758 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:47.128136 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:47.128140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:47.128143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:47.128314 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:47.128854 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:47.128860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:47.129006 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.129179 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.129422 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":416,"threads_started":1}
I20260810 19:25:47.129843 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.115621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.19.68.62:45347)
0810 19:25:47.115818 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.115826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.116492 (+ 666us) server_negotiation.cc:407] Connection header received
0810 19:25:47.116688 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.116691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.116768 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.116860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:47.118234 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.119013 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.120346 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.120823 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.120950 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.124484 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:47.124489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:47.124492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:47.124523 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:47.127178 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:47.127866 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:47.127871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:47.127874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:47.127999 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:47.128442 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:47.128445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:47.128447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:47.128728 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:47.128870 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.129292 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.129551 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":63}
I20260810 19:25:47.134924 22763 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:57052:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:25:47.136036 22763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:47.143126 23026 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=060d48190fed45edb6b52fbd3c26f334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:47.143476 23487 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=060d48190fed45edb6b52fbd3c26f334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:47.144232 23026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.144289 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.148813 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=060d48190fed45edb6b52fbd3c26f334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:25:47.149636 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ad3225dc34b8c8a2eb59e4f3688a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.151541 23662 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: Bootstrap starting.
I20260810 19:25:47.153232 23662 tablet_bootstrap.cc:654] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.153388 23661 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: Bootstrap starting.
I20260810 19:25:47.155098 23661 tablet_bootstrap.cc:654] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.155972 23662 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: No bootstrap required, opened a new log
I20260810 19:25:47.156150 23662 ts_tablet_manager.cc:1397] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 19:25:47.156204 23660 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: Bootstrap starting.
I20260810 19:25:47.157042 23662 raft_consensus.cc:348] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.157315 23662 raft_consensus.cc:374] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.157425 23662 raft_consensus.cc:729] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f34e5be0324bd2a5a2bf7155535eec, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.157711 23662 consensus_queue.cc:260] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.157920 23660 tablet_bootstrap.cc:654] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.158423 23662 ts_tablet_manager.cc:1428] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 19:25:47.158277 23672 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.158723 23672 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.160250 23660 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: No bootstrap required, opened a new log
I20260810 19:25:47.160390 23660 ts_tablet_manager.cc:1397] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 19:25:47.159569 23661 tablet_bootstrap.cc:492] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: No bootstrap required, opened a new log
I20260810 19:25:47.160849 23661 ts_tablet_manager.cc:1397] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260810 19:25:47.161108 23660 raft_consensus.cc:348] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.161303 23660 raft_consensus.cc:374] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.161398 23660 raft_consensus.cc:729] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.161623 23660 consensus_queue.cc:260] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.161798 23661 raft_consensus.cc:348] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.162077 23661 raft_consensus.cc:374] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.162115 23660 ts_tablet_manager.cc:1428] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:25:47.162036 23671 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.162197 23661 raft_consensus.cc:729] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.162361 23671 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.162470 23661 consensus_queue.cc:260] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.163172 23661 ts_tablet_manager.cc:1428] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 19:25:47.162988 23674 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.163396 23674 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.232213 23674 raft_consensus.cc:482] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:47.232556 23674 raft_consensus.cc:504] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.234144 23674 leader_election.cc:288] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:47.241221 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.234745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:54157 (local address 127.19.68.3:35855)
0810 19:25:47.234941 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.234946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.235871 (+ 925us) server_negotiation.cc:407] Connection header received
0810 19:25:47.236046 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.236052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.236117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.236192 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:47.236873 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.237693 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.239514 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.240262 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.240375 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.240470 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.240805 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.240937 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:25:47.241309 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.234519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:47489 (local address 127.19.68.2:40857)
0810 19:25:47.234713 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.234720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.235710 (+ 990us) server_negotiation.cc:407] Connection header received
0810 19:25:47.235902 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.235906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.235974 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.236049 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:25:47.236745 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.237558 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.239555 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.240396 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.240519 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.240622 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.240908 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.241034 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:25:47.241498 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.234882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:35855 (local address 127.19.68.1:54157)
0810 19:25:47.235766 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.235780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.235911 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.236228 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.236232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.236292 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:47.236704 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.236716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.237825 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.237830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.239344 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.239364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.240465 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.240471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.240631 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.240680 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.240760 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.240881 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":38,"spinlock_wait_cycles":4352,"thread_start_us":456,"threads_started":1}
I20260810 19:25:47.241631 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.234824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:40857 (local address 127.19.68.1:47489)
0810 19:25:47.235595 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.235613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.235758 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.236084 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.236089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.236113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:25:47.236521 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.236530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.237704 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.237707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.239388 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.239401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.240623 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.240629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.240767 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.240790 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.240875 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.240993 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":3,"thread_start_us":437,"threads_started":1}
I20260810 19:25:47.242214 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ad3225dc34b8c8a2eb59e4f3688a4" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" is_pre_election: true
I20260810 19:25:47.242508 23529 raft_consensus.cc:2454] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:47.242547 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ad3225dc34b8c8a2eb59e4f3688a4" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5" is_pre_election: true
I20260810 19:25:47.242981 23282 raft_consensus.cc:2454] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:47.242983 22946 leader_election.cc:302] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182555744a084d64b8a48065c49fca6e, e2f34e5be0324bd2a5a2bf7155535eec; no voters:
I20260810 19:25:47.243376 23674 raft_consensus.cc:2781] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:47.243510 23674 raft_consensus.cc:482] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:47.243613 23674 raft_consensus.cc:3037] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.245864 23674 raft_consensus.cc:504] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.246464 23674 leader_election.cc:288] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Requested vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:47.246948 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ad3225dc34b8c8a2eb59e4f3688a4" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5"
I20260810 19:25:47.246964 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ad3225dc34b8c8a2eb59e4f3688a4" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
I20260810 19:25:47.247128 23282 raft_consensus.cc:3037] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.247159 23529 raft_consensus.cc:3037] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.249110 23529 raft_consensus.cc:2454] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:47.249260 23282 raft_consensus.cc:2454] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:47.249531 22946 leader_election.cc:302] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182555744a084d64b8a48065c49fca6e, e2f34e5be0324bd2a5a2bf7155535eec; no voters:
I20260810 19:25:47.249782 23674 raft_consensus.cc:2781] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:47.250202 23674 raft_consensus.cc:686] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [term 1 LEADER]: Becoming Leader. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Running, Role: LEADER
I20260810 19:25:47.250698 23674 consensus_queue.cc:237] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.254381 23705 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: TxnStatusTablet state changed. Reason: New leader 182555744a084d64b8a48065c49fca6e. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.254722 23705 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:47.255476 23708 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:25:47.255344 22763 catalog_manager.cc:5495] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 182555744a084d64b8a48065c49fca6e (127.19.68.1). New cstate: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } health_report { overall_health: UNKNOWN } } }
I20260810 19:25:47.255753 23708 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:25:47.283244 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.277499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34121 (local address 127.0.0.1:55706)
0810 19:25:47.277792 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.277808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.277971 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.278485 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.278490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.278503 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:47.278677 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.278685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.279807 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.279812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.280916 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.280929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.281645 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.281651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.281812 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.281874 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.282694 (+ 820us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.282717 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.282797 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.282958 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260810 19:25:47.283267 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.277465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.19.68.1:34121)
0810 19:25:47.277618 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.277623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.277884 (+ 261us) server_negotiation.cc:407] Connection header received
0810 19:25:47.278178 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.278182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.278304 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.278435 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.278808 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.279658 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.281041 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.281499 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.281605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.282045 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.282589 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.282717 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.282830 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.282911 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:25:47.290625 23529 raft_consensus.cc:1261] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.290601 23282 raft_consensus.cc:1261] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.291594 23674 consensus_queue.cc:1035] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.291935 23705 consensus_queue.cc:1035] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.302724 23672 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: TxnStatusTablet state changed. Reason: New leader 182555744a084d64b8a48065c49fca6e. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.302992 23672 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.308976 23674 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.309981 23674 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:47.316522 23671 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: TxnStatusTablet state changed. Reason: New leader 182555744a084d64b8a48065c49fca6e. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.316870 23671 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.320408 23705 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.320650 23705 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 182555744a084d64b8a48065c49fca6e: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:25:47.322253 23672 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.322500 23672 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P e2f34e5be0324bd2a5a2bf7155535eec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.331538 23671 tablet_replica.cc:439] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } } }
I20260810 19:25:47.331821 23671 tablet_replica.cc:442] T 6c8ad3225dc34b8c8a2eb59e4f3688a4 P 1524942834d844a89ce0f5f6986a1dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:25:47.340888 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.334288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.19.68.2:40857)
0810 19:25:47.334448 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.334454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.335019 (+ 565us) server_negotiation.cc:407] Connection header received
0810 19:25:47.335118 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.335124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.335245 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.335397 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.335858 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.337054 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.338440 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.339106 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.339231 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.339654 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.340294 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.340439 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.340495 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.340579 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:25:47.356976 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.350883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:44359 (local address 127.19.68.1:34121)
0810 19:25:47.351081 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.351088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.351618 (+ 530us) server_negotiation.cc:407] Connection header received
0810 19:25:47.351793 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.351797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.351894 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.351968 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.352499 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.353312 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.355282 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.355764 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.355869 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.356360 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.356479 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.356544 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.356681 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.356748 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:25:47.357218 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.350847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34121 (local address 127.19.68.2:44359)
0810 19:25:47.351516 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.351536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.351681 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.351998 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.352003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.352016 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:47.352352 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.352363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.353437 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.353443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.355148 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.355163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.355930 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.355937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.356109 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.356177 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.356567 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.356579 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.356656 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.356805 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":333,"threads_started":1}
I20260810 19:25:47.367240 23063 raft_consensus.cc:1261] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Refusing update from remote peer 1524942834d844a89ce0f5f6986a1dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.367830 23529 raft_consensus.cc:1261] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Refusing update from remote peer 1524942834d844a89ce0f5f6986a1dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.367985 23678 consensus_queue.cc:1035] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.368391 23671 consensus_queue.cc:1035] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.374226 23715 mvcc.cc:205] Tried to move back new op lower bound from 7317053224409440256 to 7317053223198208000. Current Snapshot: MvccSnapshot[applied={T|T < 7317053224409440256}]
I20260810 19:25:47.410647 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.402935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:43881 (local address 127.19.68.2:40857)
0810 19:25:47.403423 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.403429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.404029 (+ 600us) server_negotiation.cc:407] Connection header received
0810 19:25:47.404273 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.404279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.404366 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.404457 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.405131 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.406269 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.408689 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.409395 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.409523 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.409900 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.410025 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.410112 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.410288 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.410373 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260810 19:25:47.410820 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.403023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:40857 (local address 127.19.68.1:43881)
0810 19:25:47.403902 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.403930 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.404090 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.404485 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.404490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.404504 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:47.404948 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.404960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.406425 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.406430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.408539 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.408556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.409561 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.409566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.409685 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.409739 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.410140 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.410163 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.410258 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.410421 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":476,"threads_started":1}
I20260810 19:25:47.449707 22766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:47.450045 22766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:47.454123 22766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Sending DeleteTablet for 3 replicas of tablet cbd0c34d3d3d42708d7bbadbc2d0b736
I20260810 19:25:47.455325 23254 tablet_service.cc:1514] Processing DeleteTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:47 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 19:25:47.455348 23487 tablet_service.cc:1514] Processing DeleteTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:47 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49234
I20260810 19:25:47.457967 23026 tablet_service.cc:1514] Processing DeleteTablet for tablet cbd0c34d3d3d42708d7bbadbc2d0b736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:47 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55700
I20260810 19:25:47.466708 22766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:47.467625 22766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:47.471335 23749 tablet_replica.cc:330] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: stopping tablet replica
I20260810 19:25:47.471997 23749 raft_consensus.cc:2229] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:47.472486 23749 raft_consensus.cc:2258] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:47.473560 23751 tablet_replica.cc:330] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: stopping tablet replica
I20260810 19:25:47.474001 23751 raft_consensus.cc:2229] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:47.474364 23751 raft_consensus.cc:2258] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:47.474959 23749 ts_tablet_manager.cc:1905] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:47.475977 23487 tablet_service.cc:1467] Processing CreateTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 (DEFAULT_TABLE table=test-table [id=e909c25b2c6d40a49023919adf0f83bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:47.476181 23751 ts_tablet_manager.cc:1905] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:47.476424 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cee87e8cc341768fdd00eb760c6e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.476928 23024 tablet_service.cc:1467] Processing CreateTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 (DEFAULT_TABLE table=test-table [id=e909c25b2c6d40a49023919adf0f83bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:47.477435 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cee87e8cc341768fdd00eb760c6e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.480253 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 (DEFAULT_TABLE table=test-table [id=e909c25b2c6d40a49023919adf0f83bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:47.480878 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cee87e8cc341768fdd00eb760c6e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:47.483623 23751 ts_tablet_manager.cc:1918] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:47.483800 23751 log.cc:1196] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/cbd0c34d3d3d42708d7bbadbc2d0b736
I20260810 19:25:47.483989 23661 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Bootstrap starting.
I20260810 19:25:47.484359 23751 ts_tablet_manager.cc:1939] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 182555744a084d64b8a48065c49fca6e: Deleting consensus metadata
I20260810 19:25:47.485662 23661 tablet_bootstrap.cc:654] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.486275 22740 catalog_manager.cc:4841] TS 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121): tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (table test-table [id=e7cc86322d474fd7acc684704de3a9a0]) successfully deleted
I20260810 19:25:47.492461 23660 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Bootstrap starting.
I20260810 19:25:47.494199 23660 tablet_bootstrap.cc:654] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.497924 23661 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: No bootstrap required, opened a new log
I20260810 19:25:47.498099 23661 ts_tablet_manager.cc:1397] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260810 19:25:47.498832 23661 raft_consensus.cc:348] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.499084 23661 raft_consensus.cc:374] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.499169 23661 raft_consensus.cc:729] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.499439 23661 consensus_queue.cc:260] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.501477 23661 ts_tablet_manager.cc:1428] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:47.504724 23748 tablet_replica.cc:330] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: stopping tablet replica
I20260810 19:25:47.505196 23748 raft_consensus.cc:2229] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:47.505498 23662 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Bootstrap starting.
I20260810 19:25:47.505858 23748 raft_consensus.cc:2258] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:47.508227 23748 ts_tablet_manager.cc:1905] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:47.509268 23662 tablet_bootstrap.cc:654] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:47.516167 23748 ts_tablet_manager.cc:1918] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:47.516347 23748 log.cc:1196] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/cbd0c34d3d3d42708d7bbadbc2d0b736
I20260810 19:25:47.516899 23748 ts_tablet_manager.cc:1939] T cbd0c34d3d3d42708d7bbadbc2d0b736 P 1524942834d844a89ce0f5f6986a1dc5: Deleting consensus metadata
I20260810 19:25:47.518682 22741 catalog_manager.cc:4841] TS 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857): tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (table test-table [id=e7cc86322d474fd7acc684704de3a9a0]) successfully deleted
I20260810 19:25:47.520318 23662 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: No bootstrap required, opened a new log
I20260810 19:25:47.520498 23662 ts_tablet_manager.cc:1397] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20260810 19:25:47.521301 23662 raft_consensus.cc:348] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.521569 23662 raft_consensus.cc:374] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.521639 23662 raft_consensus.cc:729] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f34e5be0324bd2a5a2bf7155535eec, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.521944 23662 consensus_queue.cc:260] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.523510 23662 ts_tablet_manager.cc:1428] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:47.525018 23660 tablet_bootstrap.cc:492] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: No bootstrap required, opened a new log
I20260810 19:25:47.525211 23660 ts_tablet_manager.cc:1397] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.009s
I20260810 19:25:47.526023 23660 raft_consensus.cc:348] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.526248 23660 raft_consensus.cc:374] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:47.526326 23660 raft_consensus.cc:729] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Initialized, Role: FOLLOWER
I20260810 19:25:47.526563 23660 consensus_queue.cc:260] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.527221 23660 ts_tablet_manager.cc:1428] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:25:47.531553 23749 ts_tablet_manager.cc:1918] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:47.531762 23749 log.cc:1196] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/cbd0c34d3d3d42708d7bbadbc2d0b736
I20260810 19:25:47.532260 23749 ts_tablet_manager.cc:1939] T cbd0c34d3d3d42708d7bbadbc2d0b736 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting consensus metadata
I20260810 19:25:47.534050 22746 catalog_manager.cc:4841] TS e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855): tablet cbd0c34d3d3d42708d7bbadbc2d0b736 (table test-table [id=e7cc86322d474fd7acc684704de3a9a0]) successfully deleted
I20260810 19:25:47.778787 23738 raft_consensus.cc:482] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:47.779000 23738 raft_consensus.cc:504] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.779683 23738 leader_election.cc:288] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:47.780368 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cee87e8cc341768fdd00eb760c6e48" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" is_pre_election: true
I20260810 19:25:47.780371 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cee87e8cc341768fdd00eb760c6e48" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5" is_pre_election: true
I20260810 19:25:47.780627 23529 raft_consensus.cc:2454] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:47.780643 23282 raft_consensus.cc:2454] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:47.781106 22946 leader_election.cc:302] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182555744a084d64b8a48065c49fca6e, e2f34e5be0324bd2a5a2bf7155535eec; no voters:
I20260810 19:25:47.781438 23738 raft_consensus.cc:2781] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:47.781571 23738 raft_consensus.cc:482] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:47.781673 23738 raft_consensus.cc:3037] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.784688 23738 raft_consensus.cc:504] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.785370 23738 leader_election.cc:288] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Requested vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:47.785853 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cee87e8cc341768fdd00eb760c6e48" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
I20260810 19:25:47.785876 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cee87e8cc341768fdd00eb760c6e48" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5"
I20260810 19:25:47.786055 23529 raft_consensus.cc:3037] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.786108 23282 raft_consensus.cc:3037] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:47.788028 23529 raft_consensus.cc:2454] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:47.788383 22946 leader_election.cc:302] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182555744a084d64b8a48065c49fca6e, e2f34e5be0324bd2a5a2bf7155535eec; no voters:
I20260810 19:25:47.788645 23738 raft_consensus.cc:2781] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:47.788800 23738 raft_consensus.cc:686] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 1 LEADER]: Becoming Leader. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Running, Role: LEADER
I20260810 19:25:47.789065 23738 consensus_queue.cc:237] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:47.789403 23282 raft_consensus.cc:2454] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:47.791760 22765 catalog_manager.cc:5495] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 182555744a084d64b8a48065c49fca6e (127.19.68.1). New cstate: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } health_report { overall_health: UNKNOWN } } }
I20260810 19:25:47.835404 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.828527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.19.68.1:34121)
0810 19:25:47.828734 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.828751 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.828775 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:25:47.828835 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.828840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.828922 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.829019 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.829617 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.830774 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.832767 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.833431 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.833553 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.834151 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.834283 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.834365 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.834934 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.835014 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:25:47.851470 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.846526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:38393 (local address 127.19.68.1:34121)
0810 19:25:47.846694 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:47.846699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:47.846789 (+ 90us) server_negotiation.cc:407] Connection header received
0810 19:25:47.846927 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:47.846930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:47.846989 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:47.847070 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:47.847398 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.848069 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.849862 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.850259 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.850373 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.850811 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.850928 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.850994 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:47.851123 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 19:25:47.851190 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":54}
I20260810 19:25:47.851567 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:47.846514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34121 (local address 127.19.68.1:38393)
0810 19:25:47.846702 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:47.846720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:47.846837 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:47.847082 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:47.847087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:47.847098 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:25:47.847284 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.847293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.848190 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.848195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.849709 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:47.849724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:47.850423 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:47.850429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:47.850587 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:47.850634 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:25:47.851021 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:25:47.851031 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 19:25:47.851106 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 19:25:47.851243 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:25:47.859978 23282 raft_consensus.cc:1261] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.860023 23529 raft_consensus.cc:1261] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:47.860504 23674 consensus_queue.cc:1035] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.860772 23726 consensus_queue.cc:1035] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:47.864377 23714 mvcc.cc:205] Tried to move back new op lower bound from 7317053226430427136 to 7317053226146553856. Current Snapshot: MvccSnapshot[applied={T|T < 7317053226430427136}]
W20260810 19:25:47.882412 23714 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 19:25:49.472740 22763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:49.472957 22763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:49.477376 22763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Sending DeleteTablet for 3 replicas of tablet 45cee87e8cc341768fdd00eb760c6e48
I20260810 19:25:49.478145 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55700
I20260810 19:25:49.478317 23254 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 19:25:49.478479 23487 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cee87e8cc341768fdd00eb760c6e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49234
I20260810 19:25:49.490942 23809 tablet_replica.cc:330] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: stopping tablet replica
I20260810 19:25:49.490990 23806 tablet_replica.cc:330] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: stopping tablet replica
I20260810 19:25:49.491209 23806 raft_consensus.cc:2229] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:49.491365 23809 raft_consensus.cc:2229] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:49.491746 23809 raft_consensus.cc:2258] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.491982 23806 raft_consensus.cc:2258] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.492978 23809 ts_tablet_manager.cc:1905] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:49.493031 23806 ts_tablet_manager.cc:1905] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:49.493649 23807 tablet_replica.cc:330] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: stopping tablet replica
I20260810 19:25:49.493963 23807 raft_consensus.cc:2229] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:49.494249 23807 raft_consensus.cc:2258] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.495270 23807 ts_tablet_manager.cc:1905] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:49.499547 23806 ts_tablet_manager.cc:1918] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:49.499632 23809 ts_tablet_manager.cc:1918] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:49.499747 23806 log.cc:1196] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/45cee87e8cc341768fdd00eb760c6e48
I20260810 19:25:49.499863 23809 log.cc:1196] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/45cee87e8cc341768fdd00eb760c6e48
I20260810 19:25:49.500202 23806 ts_tablet_manager.cc:1939] T 45cee87e8cc341768fdd00eb760c6e48 P 182555744a084d64b8a48065c49fca6e: Deleting consensus metadata
I20260810 19:25:49.500366 23809 ts_tablet_manager.cc:1939] T 45cee87e8cc341768fdd00eb760c6e48 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting consensus metadata
I20260810 19:25:49.501543 22740 catalog_manager.cc:4841] TS 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121): tablet 45cee87e8cc341768fdd00eb760c6e48 (table test-table [id=e909c25b2c6d40a49023919adf0f83bb]) successfully deleted
I20260810 19:25:49.501565 22746 catalog_manager.cc:4841] TS e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855): tablet 45cee87e8cc341768fdd00eb760c6e48 (table test-table [id=e909c25b2c6d40a49023919adf0f83bb]) successfully deleted
I20260810 19:25:49.502007 23807 ts_tablet_manager.cc:1918] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:49.502137 23807 log.cc:1196] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/45cee87e8cc341768fdd00eb760c6e48
I20260810 19:25:49.502581 23807 ts_tablet_manager.cc:1939] T 45cee87e8cc341768fdd00eb760c6e48 P 1524942834d844a89ce0f5f6986a1dc5: Deleting consensus metadata
I20260810 19:25:49.503576 22741 catalog_manager.cc:4841] TS 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857): tablet 45cee87e8cc341768fdd00eb760c6e48 (table test-table [id=e909c25b2c6d40a49023919adf0f83bb]) successfully deleted
Aug 10 19:25:49 dist-test-slave-2j7r krb5kdc[22693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:25:49 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:49.540408 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:49.525850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.19.68.62:45347)
0810 19:25:49.526446 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:49.526452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:49.526474 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:25:49.526623 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:49.526627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:49.526689 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:49.526891 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:49.528235 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:49.529064 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:49.530569 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:49.531013 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:49.531102 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:49.535045 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:49.535050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:49.535053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:49.535087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:49.537853 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:49.538476 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:49.538480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:49.538482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:49.538618 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:49.538999 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:49.539004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:49.539008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:49.539277 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:49.539526 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:49.539821 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 19:25:49.540052 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":319,"threads_started":1}
I20260810 19:25:49.543889 22763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:25:49.544602 22763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:49.551031 23024 tablet_service.cc:1467] Processing CreateTablet for tablet ef33719ea61148a388276003029adea6 (DEFAULT_TABLE table=test-table [id=78b9272f68cc40448778fd20215bdc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.551123 23254 tablet_service.cc:1467] Processing CreateTablet for tablet ef33719ea61148a388276003029adea6 (DEFAULT_TABLE table=test-table [id=78b9272f68cc40448778fd20215bdc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.551313 23487 tablet_service.cc:1467] Processing CreateTablet for tablet ef33719ea61148a388276003029adea6 (DEFAULT_TABLE table=test-table [id=78b9272f68cc40448778fd20215bdc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.551534 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef33719ea61148a388276003029adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.551543 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef33719ea61148a388276003029adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.551721 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef33719ea61148a388276003029adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.556762 23834 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Bootstrap starting.
I20260810 19:25:49.558915 23834 tablet_bootstrap.cc:654] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.560401 23835 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Bootstrap starting.
I20260810 19:25:49.560933 23837 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Bootstrap starting.
I20260810 19:25:49.561371 23834 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: No bootstrap required, opened a new log
I20260810 19:25:49.561518 23834 ts_tablet_manager.cc:1397] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:25:49.562314 23834 raft_consensus.cc:348] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.562561 23834 raft_consensus.cc:374] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.562667 23834 raft_consensus.cc:729] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f34e5be0324bd2a5a2bf7155535eec, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.562723 23837 tablet_bootstrap.cc:654] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.562939 23834 consensus_queue.cc:260] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.563771 23834 ts_tablet_manager.cc:1428] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:25:49.565153 23835 tablet_bootstrap.cc:654] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.566860 23837 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: No bootstrap required, opened a new log
I20260810 19:25:49.567010 23837 ts_tablet_manager.cc:1397] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 19:25:49.567554 23835 tablet_bootstrap.cc:492] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: No bootstrap required, opened a new log
I20260810 19:25:49.567684 23835 ts_tablet_manager.cc:1397] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 19:25:49.568140 23837 raft_consensus.cc:348] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.568373 23837 raft_consensus.cc:374] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.568383 23835 raft_consensus.cc:348] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.568508 23837 raft_consensus.cc:729] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.568598 23835 raft_consensus.cc:374] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.568691 23835 raft_consensus.cc:729] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.568743 23837 consensus_queue.cc:260] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.568904 23835 consensus_queue.cc:260] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.569559 23837 ts_tablet_manager.cc:1428] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 19:25:49.570035 23835 ts_tablet_manager.cc:1428] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:25:49.587183 23738 raft_consensus.cc:482] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:49.587337 23738 raft_consensus.cc:504] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.587823 23738 leader_election.cc:288] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:49.588317 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef33719ea61148a388276003029adea6" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5" is_pre_election: true
I20260810 19:25:49.588325 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef33719ea61148a388276003029adea6" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" is_pre_election: true
I20260810 19:25:49.588517 23282 raft_consensus.cc:2454] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:49.588547 23524 raft_consensus.cc:2454] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 0.
I20260810 19:25:49.588958 22943 leader_election.cc:302] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, 182555744a084d64b8a48065c49fca6e; no voters:
I20260810 19:25:49.589284 23738 raft_consensus.cc:2781] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:49.589375 23738 raft_consensus.cc:482] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:49.589461 23738 raft_consensus.cc:3037] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:49.591352 23738 raft_consensus.cc:504] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.591864 23738 leader_election.cc:288] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Requested vote from peers 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:49.592211 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef33719ea61148a388276003029adea6" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1524942834d844a89ce0f5f6986a1dc5"
I20260810 19:25:49.592291 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef33719ea61148a388276003029adea6" candidate_uuid: "182555744a084d64b8a48065c49fca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
I20260810 19:25:49.592396 23282 raft_consensus.cc:3037] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:49.592482 23524 raft_consensus.cc:3037] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:49.594724 23282 raft_consensus.cc:2454] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:49.594825 23524 raft_consensus.cc:2454] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182555744a084d64b8a48065c49fca6e in term 1.
I20260810 19:25:49.595074 22943 leader_election.cc:302] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, 182555744a084d64b8a48065c49fca6e; no voters:
I20260810 19:25:49.595324 23738 raft_consensus.cc:2781] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:49.595517 23738 raft_consensus.cc:686] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 1 LEADER]: Becoming Leader. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Running, Role: LEADER
I20260810 19:25:49.595750 23738 consensus_queue.cc:237] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.598340 22763 catalog_manager.cc:5495] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 182555744a084d64b8a48065c49fca6e (127.19.68.1). New cstate: current_term: 1 leader_uuid: "182555744a084d64b8a48065c49fca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } health_report { overall_health: UNKNOWN } } }
I20260810 19:25:49.620991 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:49.616456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.19.68.1:34121)
0810 19:25:49.616942 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:49.616946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:49.616965 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:25:49.617012 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:49.617015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:49.617072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:49.617164 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:49.617495 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:49.618245 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:49.619487 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:49.619906 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:49.619992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:49.620368 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:49.620480 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:49.620545 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:49.620673 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 19:25:49.620730 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":266,"threads_started":1}
I20260810 19:25:49.629369 23282 raft_consensus.cc:1261] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:49.629407 23524 raft_consensus.cc:1261] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Refusing update from remote peer 182555744a084d64b8a48065c49fca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:49.629951 23794 consensus_queue.cc:1035] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:49.630252 23800 consensus_queue.cc:1035] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:49.635948 23776 mvcc.cc:205] Tried to move back new op lower bound from 7317053233677365248 to 7317053233546371072. Current Snapshot: MvccSnapshot[applied={T|T < 7317053233677365248}]
I20260810 19:25:49.673614 22766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:49.673837 22766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:49.676534 22766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Sending DeleteTablet for 3 replicas of tablet ef33719ea61148a388276003029adea6
I20260810 19:25:49.677390 23254 tablet_service.cc:1514] Processing DeleteTablet for tablet ef33719ea61148a388276003029adea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 19:25:49.677393 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet ef33719ea61148a388276003029adea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55700
I20260810 19:25:49.677492 23487 tablet_service.cc:1514] Processing DeleteTablet for tablet ef33719ea61148a388276003029adea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49234
I20260810 19:25:49.677678 23807 tablet_replica.cc:330] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: stopping tablet replica
I20260810 19:25:49.677757 23809 tablet_replica.cc:330] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: stopping tablet replica
I20260810 19:25:49.677889 23807 raft_consensus.cc:2229] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:49.677955 23809 raft_consensus.cc:2229] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:49.678086 23807 raft_consensus.cc:2258] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.678179 23809 raft_consensus.cc:2258] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.678323 22766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 19:25:49.679016 23807 ts_tablet_manager.cc:1905] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:49.679054 23809 ts_tablet_manager.cc:1905] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 19:25:49.679136 22766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:25:49.680514 23806 tablet_replica.cc:330] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: stopping tablet replica
I20260810 19:25:49.680720 23806 raft_consensus.cc:2229] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:49.681039 23806 raft_consensus.cc:2258] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:49.681905 23806 ts_tablet_manager.cc:1905] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:49.685621 23809 ts_tablet_manager.cc:1918] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:49.685621 23807 ts_tablet_manager.cc:1918] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:49.685774 23807 log.cc:1196] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/ef33719ea61148a388276003029adea6
I20260810 19:25:49.685822 23809 log.cc:1196] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/ef33719ea61148a388276003029adea6
I20260810 19:25:49.686216 23809 ts_tablet_manager.cc:1939] T ef33719ea61148a388276003029adea6 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting consensus metadata
I20260810 19:25:49.686216 23807 ts_tablet_manager.cc:1939] T ef33719ea61148a388276003029adea6 P 1524942834d844a89ce0f5f6986a1dc5: Deleting consensus metadata
I20260810 19:25:49.686515 23806 ts_tablet_manager.cc:1918] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:25:49.686650 23806 log.cc:1196] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/ef33719ea61148a388276003029adea6
I20260810 19:25:49.687016 23806 ts_tablet_manager.cc:1939] T ef33719ea61148a388276003029adea6 P 182555744a084d64b8a48065c49fca6e: Deleting consensus metadata
I20260810 19:25:49.687245 22746 catalog_manager.cc:4841] TS e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855): tablet ef33719ea61148a388276003029adea6 (table test-table [id=78b9272f68cc40448778fd20215bdc9f]) successfully deleted
I20260810 19:25:49.687382 22741 catalog_manager.cc:4841] TS 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857): tablet ef33719ea61148a388276003029adea6 (table test-table [id=78b9272f68cc40448778fd20215bdc9f]) successfully deleted
I20260810 19:25:49.688349 22740 catalog_manager.cc:4841] TS 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121): tablet ef33719ea61148a388276003029adea6 (table test-table [id=78b9272f68cc40448778fd20215bdc9f]) successfully deleted
I20260810 19:25:49.692054 23024 tablet_service.cc:1467] Processing CreateTablet for tablet d5deebc90f0743269716aa0ee6c912e8 (DEFAULT_TABLE table=test-table [id=ac7b2defe24043b89cf2e163a4c019fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.692083 23254 tablet_service.cc:1467] Processing CreateTablet for tablet d5deebc90f0743269716aa0ee6c912e8 (DEFAULT_TABLE table=test-table [id=ac7b2defe24043b89cf2e163a4c019fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.692334 23487 tablet_service.cc:1467] Processing CreateTablet for tablet d5deebc90f0743269716aa0ee6c912e8 (DEFAULT_TABLE table=test-table [id=ac7b2defe24043b89cf2e163a4c019fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:25:49.692528 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5deebc90f0743269716aa0ee6c912e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.692528 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5deebc90f0743269716aa0ee6c912e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.692763 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5deebc90f0743269716aa0ee6c912e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:49.697345 23835 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Bootstrap starting.
I20260810 19:25:49.698707 23835 tablet_bootstrap.cc:654] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.699136 23837 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Bootstrap starting.
I20260810 19:25:49.700098 23834 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Bootstrap starting.
I20260810 19:25:49.700775 23837 tablet_bootstrap.cc:654] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.700841 23835 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: No bootstrap required, opened a new log
I20260810 19:25:49.700973 23835 ts_tablet_manager.cc:1397] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 19:25:49.701825 23834 tablet_bootstrap.cc:654] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:49.701838 23835 raft_consensus.cc:348] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.702121 23835 raft_consensus.cc:374] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.702216 23835 raft_consensus.cc:729] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182555744a084d64b8a48065c49fca6e, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.702468 23835 consensus_queue.cc:260] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.702958 23837 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: No bootstrap required, opened a new log
I20260810 19:25:49.703011 23835 ts_tablet_manager.cc:1428] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 19:25:49.703116 23837 ts_tablet_manager.cc:1397] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 19:25:49.703917 23834 tablet_bootstrap.cc:492] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: No bootstrap required, opened a new log
I20260810 19:25:49.703826 23837 raft_consensus.cc:348] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.704062 23837 raft_consensus.cc:374] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.704088 23834 ts_tablet_manager.cc:1397] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 19:25:49.704145 23837 raft_consensus.cc:729] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.704393 23837 consensus_queue.cc:260] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.704988 23837 ts_tablet_manager.cc:1428] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:25:49.704890 23834 raft_consensus.cc:348] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.705212 23834 raft_consensus.cc:374] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:49.705309 23834 raft_consensus.cc:729] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f34e5be0324bd2a5a2bf7155535eec, State: Initialized, Role: FOLLOWER
I20260810 19:25:49.705559 23834 consensus_queue.cc:260] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:49.706302 23834 ts_tablet_manager.cc:1428] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 19:25:50.032572 23842 raft_consensus.cc:482] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:25:50.032816 23842 raft_consensus.cc:504] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:50.033576 23842 leader_election.cc:288] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:50.034192 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5deebc90f0743269716aa0ee6c912e8" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182555744a084d64b8a48065c49fca6e" is_pre_election: true
I20260810 19:25:50.034194 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5deebc90f0743269716aa0ee6c912e8" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" is_pre_election: true
I20260810 19:25:50.034431 23063 raft_consensus.cc:2454] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 0.
I20260810 19:25:50.034452 23524 raft_consensus.cc:2454] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 0.
I20260810 19:25:50.034876 23184 leader_election.cc:302] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, e2f34e5be0324bd2a5a2bf7155535eec; no voters:
I20260810 19:25:50.035156 23842 raft_consensus.cc:2781] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:25:50.035275 23842 raft_consensus.cc:482] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:25:50.035365 23842 raft_consensus.cc:3037] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:50.037621 23842 raft_consensus.cc:504] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:50.038146 23842 leader_election.cc:288] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121), e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855)
I20260810 19:25:50.038563 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5deebc90f0743269716aa0ee6c912e8" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182555744a084d64b8a48065c49fca6e"
I20260810 19:25:50.038666 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5deebc90f0743269716aa0ee6c912e8" candidate_uuid: "1524942834d844a89ce0f5f6986a1dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f34e5be0324bd2a5a2bf7155535eec"
I20260810 19:25:50.038743 23063 raft_consensus.cc:3037] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:50.038858 23524 raft_consensus.cc:3037] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:50.040870 23524 raft_consensus.cc:2454] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 1.
I20260810 19:25:50.040881 23063 raft_consensus.cc:2454] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524942834d844a89ce0f5f6986a1dc5 in term 1.
I20260810 19:25:50.041301 23180 leader_election.cc:302] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524942834d844a89ce0f5f6986a1dc5, 182555744a084d64b8a48065c49fca6e; no voters:
I20260810 19:25:50.041605 23842 raft_consensus.cc:2781] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:50.041975 23842 raft_consensus.cc:686] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1524942834d844a89ce0f5f6986a1dc5, State: Running, Role: LEADER
I20260810 19:25:50.042253 23842 consensus_queue.cc:237] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } }
I20260810 19:25:50.044647 22766 catalog_manager.cc:5495] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "1524942834d844a89ce0f5f6986a1dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1524942834d844a89ce0f5f6986a1dc5" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 40857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 } health_report { overall_health: UNKNOWN } } }
I20260810 19:25:50.081770 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:50.072633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.19.68.2:40857)
0810 19:25:50.077474 (+ 4841us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:50.077478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:50.077496 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:25:50.077545 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:50.077548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:50.077605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:50.077678 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:50.078083 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:50.078829 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:50.080155 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:50.080551 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:50.080640 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:50.081062 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:50.081204 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:50.081270 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:50.081394 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 19:25:50.081454 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4737,"thread_start_us":4619,"threads_started":1}
I20260810 19:25:50.089962 23524 raft_consensus.cc:1261] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Refusing update from remote peer 1524942834d844a89ce0f5f6986a1dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:50.090124 23063 raft_consensus.cc:1261] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Refusing update from remote peer 1524942834d844a89ce0f5f6986a1dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:50.090562 23866 consensus_queue.cc:1035] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f34e5be0324bd2a5a2bf7155535eec" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:50.090844 23842 consensus_queue.cc:1035] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182555744a084d64b8a48065c49fca6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:25:50.110666 23781 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 19:25:51.690680 22766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:51.690917 22766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:25:51.695066 22766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77abdc4eb7734bf6a3da9ccd55b613ff: Sending DeleteTablet for 3 replicas of tablet d5deebc90f0743269716aa0ee6c912e8
I20260810 19:25:51.695814 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet d5deebc90f0743269716aa0ee6c912e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55700
I20260810 19:25:51.695878 23254 tablet_service.cc:1514] Processing DeleteTablet for tablet d5deebc90f0743269716aa0ee6c912e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 19:25:51.695990 23487 tablet_service.cc:1514] Processing DeleteTablet for tablet d5deebc90f0743269716aa0ee6c912e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:25:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49234
I20260810 19:25:51.696691 23924 tablet_replica.cc:330] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: stopping tablet replica
I20260810 19:25:51.696834 23922 tablet_replica.cc:330] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: stopping tablet replica
I20260810 19:25:51.696969 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22903
I20260810 19:25:51.696987 23924 raft_consensus.cc:2229] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:51.697031 23922 raft_consensus.cc:2229] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:25:51.697227 23924 raft_consensus.cc:2258] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:51.697355 23922 raft_consensus.cc:2258] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:51.697893 23921 tablet_replica.cc:330] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: stopping tablet replica
I20260810 19:25:51.698068 23921 raft_consensus.cc:2229] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:25:51.698284 23921 raft_consensus.cc:2258] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:25:51.698534 23924 ts_tablet_manager.cc:1905] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:51.699143 23921 ts_tablet_manager.cc:1905] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:51.699359 23922 ts_tablet_manager.cc:1905] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:25:51.704870 23924 ts_tablet_manager.cc:1918] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:51.705044 23924 log.cc:1196] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/d5deebc90f0743269716aa0ee6c912e8
I20260810 19:25:51.705426 23921 ts_tablet_manager.cc:1918] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:51.705562 23924 ts_tablet_manager.cc:1939] T d5deebc90f0743269716aa0ee6c912e8 P e2f34e5be0324bd2a5a2bf7155535eec: Deleting consensus metadata
I20260810 19:25:51.705595 23921 log.cc:1196] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/d5deebc90f0743269716aa0ee6c912e8
I20260810 19:25:51.706080 23921 ts_tablet_manager.cc:1939] T d5deebc90f0743269716aa0ee6c912e8 P 182555744a084d64b8a48065c49fca6e: Deleting consensus metadata
I20260810 19:25:51.706681 22746 catalog_manager.cc:4841] TS e2f34e5be0324bd2a5a2bf7155535eec (127.19.68.3:35855): tablet d5deebc90f0743269716aa0ee6c912e8 (table test-table [id=ac7b2defe24043b89cf2e163a4c019fc]) successfully deleted
I20260810 19:25:51.707374 22740 catalog_manager.cc:4841] TS 182555744a084d64b8a48065c49fca6e (127.19.68.1:34121): tablet d5deebc90f0743269716aa0ee6c912e8 (table test-table [id=ac7b2defe24043b89cf2e163a4c019fc]) successfully deleted
I20260810 19:25:51.707595 23922 ts_tablet_manager.cc:1918] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:25:51.707738 23922 log.cc:1196] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/d5deebc90f0743269716aa0ee6c912e8
I20260810 19:25:51.708230 23922 ts_tablet_manager.cc:1939] T d5deebc90f0743269716aa0ee6c912e8 P 1524942834d844a89ce0f5f6986a1dc5: Deleting consensus metadata
I20260810 19:25:51.709075 22741 catalog_manager.cc:4841] TS 1524942834d844a89ce0f5f6986a1dc5 (127.19.68.2:40857): tablet d5deebc90f0743269716aa0ee6c912e8 (table test-table [id=ac7b2defe24043b89cf2e163a4c019fc]) successfully deleted
Aug 10 19:25:51 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:25:51.720736 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:51.699799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.19.68.1:34121)
0810 19:25:51.707316 (+ 7517us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:51.707321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:51.707809 (+ 488us) server_negotiation.cc:407] Connection header received
0810 19:25:51.707989 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:51.707995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:51.708073 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:51.708195 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:51.709432 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.710156 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.711297 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.711679 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.711779 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:51.715533 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:51.715546 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:51.715563 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:51.715592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:51.718330 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.718897 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.718904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.718906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.719028 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.719370 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.719374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.719376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.719732 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:51.719995 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:51.720163 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 19:25:51.720381 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7382,"thread_start_us":7285,"threads_started":1}
W20260810 19:25:51.721383 23141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55722
W20260810 19:25:51.721987 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:51.722187 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22903
I20260810 19:25:51.760676 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23155
Aug 10 19:25:51 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:25:51.778153 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:51.761435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.19.68.2:40857)
0810 19:25:51.761923 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:51.761929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:51.761946 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:25:51.762000 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:51.762004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:51.762064 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:51.762151 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:51.763440 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.764175 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.765592 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.765988 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.766208 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:51.773070 (+ 6862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:51.773111 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:51.773130 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:51.773171 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:51.775733 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.776306 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.776312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.776315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.776430 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.776691 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.776694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.776695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.777040 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:51.777424 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:51.777641 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 19:25:51.777810 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
W20260810 19:25:51.778792 23360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612
W20260810 19:25:51.779392 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23155: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:51.779556 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23155
I20260810 19:25:51.806957 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23372
Aug 10 19:25:51 dist-test-slave-2j7r krb5kdc[22693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:25:51.823189 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:51.807683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.19.68.3:35855)
0810 19:25:51.808280 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:51.808287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:51.808306 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:25:51.808365 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:51.808371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:51.808446 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:51.808553 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:51.809836 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.810991 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.812132 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:51.812849 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:51.812977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:51.817397 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:51.817414 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:51.817430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:51.817466 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:51.820462 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.821284 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.821294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.821297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.821434 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:51.821797 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:51.821800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:51.821802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:51.822182 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:51.822443 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:51.822717 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 19:25:51.822900 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":285,"threads_started":1}
W20260810 19:25:51.823836 23616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49240
W20260810 19:25:51.824450 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23372: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:51.824592 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23372
I20260810 19:25:51.851948 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22714
I20260810 19:25:51.853166 22836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:51.970336 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22714
2026-08-10T19:25:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8648 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:52 dist-test-slave-2j7r krb5kdc[23946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:52 dist-test-slave-2j7r krb5kdc[23946](info): set up 2 sockets
Aug 10 19:25:52 dist-test-slave-2j7r krb5kdc[23946](info): commencing operation
krb5kdc: starting...
W20260810 19:25:53.199837 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:53 dist-test-slave-2j7r krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:53.412747 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37047
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:53.736403 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:53.736709 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:53.736873 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:53.744267 23974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:53.744376 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:53.744459 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:53.744553 23974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:53.744619 23974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:53.754958 23974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:37047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:53.756218 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:53.757515 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:53.769591 23981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:53.769526 23984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:53.769629 23982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:53.770220 23974 server_base.cc:1034] running on GCE node
I20260810 19:25:53.771059 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:53.772429 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:53.773646 23974 hybrid_clock.cc:648] HybridClock initialized: now 1786389953773589 us; error 40 us; skew 500 ppm
I20260810 19:25:53.773967 23974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:53 dist-test-slave-2j7r krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389953, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:53.777124 23974 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:53.778558 23974 webserver.cc:460] Webserver started at http://127.19.68.62:33705/ using document root <none> and password file <none>
I20260810 19:25:53.779181 23974 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:53.779294 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:53.779567 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:53.782513 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "435eed01c16747aa8566938d206347f3"
format_stamp: "Formatted at 2026-08-10 19:25:53 on dist-test-slave-2j7r"
server_key: "9414f45316cfd648c2e97166f59f4c1e"
server_key_iv: "e11497d2deacbd9334f913ac0a2d3724"
server_key_version: "encryptionkey@0"
I20260810 19:25:53.783215 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "435eed01c16747aa8566938d206347f3"
format_stamp: "Formatted at 2026-08-10 19:25:53 on dist-test-slave-2j7r"
server_key: "9414f45316cfd648c2e97166f59f4c1e"
server_key_iv: "e11497d2deacbd9334f913ac0a2d3724"
server_key_version: "encryptionkey@0"
I20260810 19:25:53.788096 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:53.791401 23999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:53.792709 23974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:25:53.792861 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "435eed01c16747aa8566938d206347f3"
format_stamp: "Formatted at 2026-08-10 19:25:53 on dist-test-slave-2j7r"
server_key: "9414f45316cfd648c2e97166f59f4c1e"
server_key_iv: "e11497d2deacbd9334f913ac0a2d3724"
server_key_version: "encryptionkey@0"
I20260810 19:25:53.793030 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:53.818938 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:53.823022 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:53.823261 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:53.840481 23974 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:37047
I20260810 19:25:53.840514 24087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:37047 every 8 connection(s)
I20260810 19:25:53.841804 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:53.842553 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23974
I20260810 19:25:53.842806 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:53.843523 19728 external_mini_cluster.cc:1428] Setting key be3ede793ce5fc62e8c35b4cdfb56634
I20260810 19:25:53.846185 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:25:53 dist-test-slave-2j7r krb5kdc[23946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:53.858748 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3: Bootstrap starting.
I20260810 19:25:53.862146 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:53.863350 24088 log.cc:824] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:53.866640 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3: No bootstrap required, opened a new log
I20260810 19:25:53.866838 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:53.845370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.19.68.62:37047)
0810 19:25:53.846153 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:53.846168 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:53.846224 (+ 56us) server_negotiation.cc:407] Connection header received
0810 19:25:53.848539 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:53.848567 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:53.849020 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:53.849485 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:53.850772 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:53.852809 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:53.854096 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:53.855078 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:53.855258 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:53.859367 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:53.859375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:53.859388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:53.859428 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:53.862412 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:53.863217 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:53.863223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:53.863228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:53.863407 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:53.863792 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:53.863796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:53.863798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:53.864322 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:53.864576 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:53.865067 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 19:25:53.865516 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":304,"threads_started":1}
I20260810 19:25:53.874305 24088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } }
I20260810 19:25:53.874622 24088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:53.874693 24088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435eed01c16747aa8566938d206347f3, State: Initialized, Role: FOLLOWER
I20260810 19:25:53.875154 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } }
I20260810 19:25:53.875339 24088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:53.875432 24088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:53.875551 24088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:53.877585 24088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } }
I20260810 19:25:53.878026 24088 leader_election.cc:302] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 435eed01c16747aa8566938d206347f3; no voters:
I20260810 19:25:53.878546 24088 leader_election.cc:288] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:53.878669 24097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:53.879709 24097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [term 1 LEADER]: Becoming Leader. State: Replica: 435eed01c16747aa8566938d206347f3, State: Running, Role: LEADER
I20260810 19:25:53.880268 24097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } }
I20260810 19:25:53.880733 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:53.883180 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 435eed01c16747aa8566938d206347f3. Latest consensus state: current_term: 1 leader_uuid: "435eed01c16747aa8566938d206347f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } } }
I20260810 19:25:53.883258 24098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "435eed01c16747aa8566938d206347f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435eed01c16747aa8566938d206347f3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37047 } } }
I20260810 19:25:53.883569 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:53.883615 24098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:53.889714 24118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:53.894630 24118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:53.904389 24118 catalog_manager.cc:1347] Generated new cluster ID: a0ab1a14b48647ad93f827133776f026
I20260810 19:25:53.904515 24118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:53.912714 24118 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:53.914026 24118 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:53.929525 24118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 435eed01c16747aa8566938d206347f3: Generated new TSK 0
I20260810 19:25:53.930342 24118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:53.980608 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:54.302436 24139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:54.302704 24139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:54.302855 24139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:54.309939 24139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:54.310038 24139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:54.310194 24139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:25:54.319692 24139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:54.320946 24139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:54.322290 24139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:54.335353 24148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:54.335417 24150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:54.335441 24156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:54.336185 24139 server_base.cc:1034] running on GCE node
I20260810 19:25:54.336731 24139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:54.337683 24139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:54.338883 24139 hybrid_clock.cc:648] HybridClock initialized: now 1786389954338829 us; error 42 us; skew 500 ppm
I20260810 19:25:54.339183 24139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:54 dist-test-slave-2j7r krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389954, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:54.342157 24139 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:25:54.343523 24139 webserver.cc:460] Webserver started at http://127.19.68.1:38725/ using document root <none> and password file <none>
I20260810 19:25:54.344180 24139 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:54.344296 24139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:54.344596 24139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:54.347512 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "1b5a47038f954aae925b547ee40cae0e"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "2e9bdfe1ff991ff39d0297c0a7df09fd"
server_key_iv: "4d42da7a4c634bf2f2fdcbc1bea5840c"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.348289 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "1b5a47038f954aae925b547ee40cae0e"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "2e9bdfe1ff991ff39d0297c0a7df09fd"
server_key_iv: "4d42da7a4c634bf2f2fdcbc1bea5840c"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.353363 24139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:25:54.356565 24165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.357825 24139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:54.357987 24139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "1b5a47038f954aae925b547ee40cae0e"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "2e9bdfe1ff991ff39d0297c0a7df09fd"
server_key_iv: "4d42da7a4c634bf2f2fdcbc1bea5840c"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.358167 24139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:54.376672 24139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:54.380504 24139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:54.380779 24139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:54.381738 24139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:54.383147 24139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:54.383219 24139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.383315 24139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:54.383358 24139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.412223 24139 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:36305
I20260810 19:25:54.412266 24357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:36305 every 8 connection(s)
I20260810 19:25:54.413506 24139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:25:54.420648 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24139
I20260810 19:25:54.420876 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:25:54.421756 19728 external_mini_cluster.cc:1428] Setting key 04b1f5cbd5b335d9b728bdea8df523d7
Aug 10 19:25:54 dist-test-slave-2j7r krb5kdc[23946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389954, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:54.433660 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:54.415963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:42841 (local address 127.19.68.62:37047)
0810 19:25:54.416399 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:54.416403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:54.417378 (+ 975us) server_negotiation.cc:407] Connection header received
0810 19:25:54.419106 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:54.419111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:54.419179 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:54.419275 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:54.420665 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.421767 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.423485 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.423980 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.424104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:54.427878 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:54.427883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:54.427886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:54.427919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:54.430571 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:54.431384 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:54.431390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:54.431393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:54.431565 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:54.432036 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:54.432040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:54.432043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:54.432400 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:54.432545 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:54.433186 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 19:25:54.433383 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
I20260810 19:25:54.434898 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:54.416501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37047 (local address 127.19.68.1:42841)
0810 19:25:54.417232 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:54.417270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:54.418843 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:54.419451 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:54.419458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:54.419964 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:54.420476 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:54.420493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.421939 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.421943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:54.423311 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:54.423327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.424227 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.424235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:54.424466 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:54.425124 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:54.425160 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:54.427647 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:54.430710 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:54.430718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:54.430732 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:54.431259 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:54.431682 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:54.431686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:54.431689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:54.431931 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:54.432508 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:54.432514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:54.432824 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 19:25:54.433051 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 19:25:54.433580 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":268,"threads_started":1}
I20260810 19:25:54.436540 24359 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37047
I20260810 19:25:54.436820 24359 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:54.437588 24359 heartbeater.cc:511] Master 127.19.68.62:37047 requested a full tablet report, sending...
I20260810 19:25:54.439574 24027 ts_manager.cc:194] Registered new tserver with Master: 1b5a47038f954aae925b547ee40cae0e (127.19.68.1:36305)
I20260810 19:25:54.441054 24027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:42841
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:25:54.472185 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:54.793987 24369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:54.794301 24369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:54.794459 24369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:54.802129 24369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:54.802271 24369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:54.802450 24369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:25:54.812628 24369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:54.814044 24369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:54.815507 24369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:54.828806 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:54.828801 24382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:54.829437 24369 server_base.cc:1034] running on GCE node
W20260810 19:25:54.829491 24384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:54.830133 24369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:54.831038 24369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:54.832243 24369 hybrid_clock.cc:648] HybridClock initialized: now 1786389954832196 us; error 34 us; skew 500 ppm
I20260810 19:25:54.832578 24369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:54 dist-test-slave-2j7r krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389954, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:54.835548 24369 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:25:54.836892 24369 webserver.cc:460] Webserver started at http://127.19.68.2:38061/ using document root <none> and password file <none>
I20260810 19:25:54.837587 24369 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:54.837702 24369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:54.837994 24369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:54.840873 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "f2969b914d3d4b2dbb643ae56156d682"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "8e9a6afdf39ac030b24df724ec491eed"
server_key_iv: "ef15dca89753e9e471fd65aa9e34914e"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.841703 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "f2969b914d3d4b2dbb643ae56156d682"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "8e9a6afdf39ac030b24df724ec491eed"
server_key_iv: "ef15dca89753e9e471fd65aa9e34914e"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.846721 24369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:25:54.850266 24397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.851603 24369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:54.851758 24369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "f2969b914d3d4b2dbb643ae56156d682"
format_stamp: "Formatted at 2026-08-10 19:25:54 on dist-test-slave-2j7r"
server_key: "8e9a6afdf39ac030b24df724ec491eed"
server_key_iv: "ef15dca89753e9e471fd65aa9e34914e"
server_key_version: "encryptionkey@0"
I20260810 19:25:54.851930 24369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:54.873327 24369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:54.876994 24369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:54.877269 24369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:54.878170 24369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:54.879480 24369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:54.879562 24369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.879678 24369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:54.879732 24369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:54.909327 24369 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:38165
I20260810 19:25:54.909370 24597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:38165 every 8 connection(s)
I20260810 19:25:54.910578 24369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:25:54.912822 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24369
I20260810 19:25:54.912997 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:25:54.913554 19728 external_mini_cluster.cc:1428] Setting key a4b040d7d9b0ea1a9867dd0ec66334c7
Aug 10 19:25:54 dist-test-slave-2j7r krb5kdc[23946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389954, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:54.930097 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:54.913137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:58401 (local address 127.19.68.62:37047)
0810 19:25:54.913376 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:54.913383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:54.914558 (+ 1175us) server_negotiation.cc:407] Connection header received
0810 19:25:54.916383 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:54.916389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:54.916465 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:54.916564 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:54.918042 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.918983 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.920442 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.920997 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.921133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:54.924559 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:54.924564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:54.924567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:54.924601 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:54.927090 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:54.927774 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:54.927779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:54.927781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:54.927921 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:54.928326 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:54.928330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:54.928332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:54.928586 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:54.928741 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:54.929434 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 19:25:54.929665 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:54.930975 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:54.913600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37047 (local address 127.19.68.2:58401)
0810 19:25:54.914405 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:54.914459 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:54.916130 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:54.916755 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:54.916763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:54.917329 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:54.917877 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:54.917899 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.919106 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.919110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:54.920307 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:54.920318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:54.921200 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:54.921205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:54.921394 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:54.921981 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:54.922004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:54.924319 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:54.927220 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:54.927229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:54.927244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:54.927655 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:54.928024 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:54.928029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:54.928034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:54.928214 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:54.928710 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:54.928718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:54.929035 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 19:25:54.929307 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 19:25:54.929844 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":317,"threads_started":1}
I20260810 19:25:54.932368 24599 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37047
I20260810 19:25:54.932646 24599 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:54.933374 24599 heartbeater.cc:511] Master 127.19.68.62:37047 requested a full tablet report, sending...
I20260810 19:25:54.934885 24027 ts_manager.cc:194] Registered new tserver with Master: f2969b914d3d4b2dbb643ae56156d682 (127.19.68.2:38165)
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
I20260810 19:25:54.935824 24027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:58401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:25:54.961882 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:55.296182 24610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:55.296494 24610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:55.296602 24610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:55.303947 24610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:55.304062 24610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:55.304191 24610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:25:55.313897 24610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:55.315235 24610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:55.316532 24610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:55.331055 24618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:55.331079 24621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:55.331462 24617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:55.338536 24610 server_base.cc:1034] running on GCE node
I20260810 19:25:55.339126 24610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:55.340041 24610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:55.341207 24610 hybrid_clock.cc:648] HybridClock initialized: now 1786389955341155 us; error 31 us; skew 500 ppm
I20260810 19:25:55.341516 24610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:55 dist-test-slave-2j7r krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389955, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:55.344650 24610 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:25:55.346113 24610 webserver.cc:460] Webserver started at http://127.19.68.3:43135/ using document root <none> and password file <none>
I20260810 19:25:55.346781 24610 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:55.346877 24610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:55.347131 24610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:55.350162 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "82a909efd1314bfdb1d13914b8e94a23"
format_stamp: "Formatted at 2026-08-10 19:25:55 on dist-test-slave-2j7r"
server_key: "fd4c75cd4af26e245e40fb41de83e74c"
server_key_iv: "8ac9d959cf81740558d21a9e0cea83f3"
server_key_version: "encryptionkey@0"
I20260810 19:25:55.350895 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "82a909efd1314bfdb1d13914b8e94a23"
format_stamp: "Formatted at 2026-08-10 19:25:55 on dist-test-slave-2j7r"
server_key: "fd4c75cd4af26e245e40fb41de83e74c"
server_key_iv: "8ac9d959cf81740558d21a9e0cea83f3"
server_key_version: "encryptionkey@0"
I20260810 19:25:55.355799 24610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:55.359093 24639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:55.360327 24610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:55.360462 24610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "82a909efd1314bfdb1d13914b8e94a23"
format_stamp: "Formatted at 2026-08-10 19:25:55 on dist-test-slave-2j7r"
server_key: "fd4c75cd4af26e245e40fb41de83e74c"
server_key_iv: "8ac9d959cf81740558d21a9e0cea83f3"
server_key_version: "encryptionkey@0"
I20260810 19:25:55.360625 24610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:55.406314 24610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:55.410481 24610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:55.410732 24610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:55.411698 24610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:25:55.413182 24610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:25:55.413264 24610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:55.413388 24610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:25:55.413443 24610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:55.443687 24610 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:37053
I20260810 19:25:55.443729 24826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:37053 every 8 connection(s)
I20260810 19:25:55.443819 24359 heartbeater.cc:503] Master 127.19.68.62:37047 was elected leader, sending a full tablet report...
I20260810 19:25:55.445183 24610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:25:55.453001 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24610
I20260810 19:25:55.453209 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:25:55.453637 19728 external_mini_cluster.cc:1428] Setting key d7665fe760d8440e746ad16bf4a9cd66
Aug 10 19:25:55 dist-test-slave-2j7r krb5kdc[23946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389955, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:55.467701 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:55.447796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:52765 (local address 127.19.68.62:37047)
0810 19:25:55.448279 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:55.448285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:55.449367 (+ 1082us) server_negotiation.cc:407] Connection header received
0810 19:25:55.451302 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:55.451309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:55.451390 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:55.451495 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:55.453053 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:55.454009 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:55.457650 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:55.458338 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:55.458474 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:55.462371 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:55.462375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:55.462379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:55.462411 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:55.464923 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:55.465657 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:55.465661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:55.465664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:55.465799 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:55.466225 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:55.466228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:55.466230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:55.466535 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:55.466644 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:55.467226 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 19:25:55.467433 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260810 19:25:55.468606 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:55.448363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37047 (local address 127.19.68.3:52765)
0810 19:25:55.449191 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:55.449251 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:55.451032 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:55.451757 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:55.451765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:55.452315 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:55.452873 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:55.452892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:55.454172 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:55.454176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:55.457478 (+ 3302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:55.457494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:55.458598 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:55.458605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:55.458927 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:55.459671 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:55.459703 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:55.462171 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:55.465044 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:55.465050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:55.465063 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:55.465542 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:55.465919 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:55.465923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:55.465925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:55.466135 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:55.466629 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:55.466634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:55.466941 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 19:25:55.467157 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 19:25:55.467520 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":313,"threads_started":1}
I20260810 19:25:55.470214 24827 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37047
I20260810 19:25:55.470548 24827 heartbeater.cc:464] Registering TS with master...
I20260810 19:25:55.471406 24827 heartbeater.cc:511] Master 127.19.68.62:37047 requested a full tablet report, sending...
I20260810 19:25:55.472940 24027 ts_manager.cc:194] Registered new tserver with Master: 82a909efd1314bfdb1d13914b8e94a23 (127.19.68.3:37053)
I20260810 19:25:55.473807 24027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:52765
I20260810 19:25:55.483608 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:25:55.521561 24832 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:55.519099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.19.68.62:37047)
0810 19:25:55.519292 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:55.519297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:55.519768 (+ 471us) server_negotiation.cc:407] Connection header received
0810 19:25:55.519927 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:55.519931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:55.520000 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:55.520092 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:55.521317 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51322: BlockingRecv error: recv got EOF from 127.0.0.1:51322 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:25:55.524580 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24139
W20260810 19:25:55.527292 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24139: Not authorized: Client connection negotiation failed: client connection to 127.19.68.1:36305: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:25:55.527441 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24139
I20260810 19:25:55.549952 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24369
W20260810 19:25:55.552641 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24369: Not authorized: Client connection negotiation failed: client connection to 127.19.68.2:38165: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:25:55.552822 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24369
I20260810 19:25:55.579080 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24610
W20260810 19:25:55.581903 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24610: Not authorized: Client connection negotiation failed: client connection to 127.19.68.3:37053: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:25:55.582065 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24610
I20260810 19:25:55.606125 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23974
I20260810 19:25:55.607304 24080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:55.729223 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23974
2026-08-10T19:25:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3752 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:55 dist-test-slave-2j7r krb5kdc[24862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:55 dist-test-slave-2j7r krb5kdc[24862](info): set up 2 sockets
Aug 10 19:25:55 dist-test-slave-2j7r krb5kdc[24862](info): commencing operation
krb5kdc: starting...
W20260810 19:25:56.956365 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[24862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:57.178671 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44545
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:44545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:57.501169 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:57.501468 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:57.501596 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:57.508842 24878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:57.508944 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:57.509018 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:57.509109 24878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:57.509174 24878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:57.518980 24878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:44545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:57.520262 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:57.521906 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:57.533653 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:57.533617 24886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:57.533842 24885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:57.534324 24878 server_base.cc:1034] running on GCE node
I20260810 19:25:57.535073 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:57.536473 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:57.537679 24878 hybrid_clock.cc:648] HybridClock initialized: now 1786389957537634 us; error 35 us; skew 500 ppm
I20260810 19:25:57.538028 24878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[24862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389957, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:57.544845 24878 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:57.546391 24878 webserver.cc:460] Webserver started at http://127.19.68.62:36991/ using document root <none> and password file <none>
I20260810 19:25:57.547068 24878 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:57.547189 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:57.547498 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:57.550549 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "d287224d4a7f40228609c0cb84fbfcda"
format_stamp: "Formatted at 2026-08-10 19:25:57 on dist-test-slave-2j7r"
server_key: "60a49bb854f7857549a73e1c09a53799"
server_key_iv: "9aafecd0f174b2431cd66e27341b855c"
server_key_version: "encryptionkey@0"
I20260810 19:25:57.551319 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "d287224d4a7f40228609c0cb84fbfcda"
format_stamp: "Formatted at 2026-08-10 19:25:57 on dist-test-slave-2j7r"
server_key: "60a49bb854f7857549a73e1c09a53799"
server_key_iv: "9aafecd0f174b2431cd66e27341b855c"
server_key_version: "encryptionkey@0"
I20260810 19:25:57.556396 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260810 19:25:57.559724 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:57.560984 24878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:25:57.561185 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "d287224d4a7f40228609c0cb84fbfcda"
format_stamp: "Formatted at 2026-08-10 19:25:57 on dist-test-slave-2j7r"
server_key: "60a49bb854f7857549a73e1c09a53799"
server_key_iv: "9aafecd0f174b2431cd66e27341b855c"
server_key_version: "encryptionkey@0"
I20260810 19:25:57.561357 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:57.582304 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:57.586324 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:57.586596 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:57.604755 24878 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:44545
I20260810 19:25:57.604787 24991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:44545 every 8 connection(s)
I20260810 19:25:57.606181 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:57.608722 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24878
I20260810 19:25:57.608975 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:57.609838 19728 external_mini_cluster.cc:1428] Setting key 4a8eb1927eddaf5f638d1436238f1db3
I20260810 19:25:57.610627 24992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[24862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:57.625201 24992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda: Bootstrap starting.
I20260810 19:25:57.629446 24992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:57.630668 24992 log.cc:824] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:57.634023 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:57.611750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.19.68.62:44545)
0810 19:25:57.612540 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:57.612555 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:57.612597 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:25:57.614205 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:57.614227 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:57.614606 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:57.614945 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:57.616819 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:57.618256 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:57.619906 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:57.620546 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:57.620686 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:57.625499 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:57.625514 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:57.625533 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:57.625587 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:57.629355 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:57.630054 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:57.630062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:57.630069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:57.630286 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:57.630640 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:57.630645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:57.630648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:57.631450 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:57.631687 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:57.632219 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 19:25:57.632685 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":340,"threads_started":1}
I20260810 19:25:57.634197 24992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda: No bootstrap required, opened a new log
I20260810 19:25:57.637842 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:57.641991 24992 raft_consensus.cc:348] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } }
I20260810 19:25:57.642347 24992 raft_consensus.cc:374] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:57.642436 24992 raft_consensus.cc:729] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d287224d4a7f40228609c0cb84fbfcda, State: Initialized, Role: FOLLOWER
I20260810 19:25:57.642995 24992 consensus_queue.cc:260] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } }
I20260810 19:25:57.643224 24992 raft_consensus.cc:388] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:57.643349 24992 raft_consensus.cc:482] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:57.643499 24992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:57.645663 24992 raft_consensus.cc:504] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } }
I20260810 19:25:57.646150 24992 leader_election.cc:302] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d287224d4a7f40228609c0cb84fbfcda; no voters:
I20260810 19:25:57.646754 24992 leader_election.cc:288] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:57.646891 24998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:57.647979 24998 raft_consensus.cc:686] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [term 1 LEADER]: Becoming Leader. State: Replica: d287224d4a7f40228609c0cb84fbfcda, State: Running, Role: LEADER
I20260810 19:25:57.648483 24998 consensus_queue.cc:237] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } }
I20260810 19:25:57.648926 24992 sys_catalog.cc:564] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:57.651082 25003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [sys.catalog]: SysCatalogTable state changed. Reason: New leader d287224d4a7f40228609c0cb84fbfcda. Latest consensus state: current_term: 1 leader_uuid: "d287224d4a7f40228609c0cb84fbfcda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } } }
I20260810 19:25:57.651458 25003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:57.651928 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d287224d4a7f40228609c0cb84fbfcda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287224d4a7f40228609c0cb84fbfcda" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44545 } } }
I20260810 19:25:57.652227 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:57.661010 25011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:57.666777 25011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:57.678952 25011 catalog_manager.cc:1347] Generated new cluster ID: 002384fa90fb41779404dca26fa98894
I20260810 19:25:57.679180 25011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:57.696506 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:57.687570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.19.68.62:44545)
0810 19:25:57.687737 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:57.687743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:57.688293 (+ 550us) server_negotiation.cc:407] Connection header received
0810 19:25:57.688466 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:57.688472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:57.688564 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:57.688677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:57.689833 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:57.691006 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:57.692768 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:57.693488 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:57.693635 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:57.694746 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:57.694753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:57.694758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:57.695653 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:57.695714 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:57.695778 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:57.695890 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:57.696020 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 19:25:57.696132 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:25:57.713574 25011 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:57.715459 25011 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:57.728318 25011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d287224d4a7f40228609c0cb84fbfcda: Generated new TSK 0
I20260810 19:25:57.729363 25011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:57.741245 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24878
I20260810 19:25:57.742624 24986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:57.833959 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24878
2026-08-10T19:25:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2100 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[25043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[25043](info): set up 2 sockets
Aug 10 19:25:57 dist-test-slave-2j7r krb5kdc[25043](info): commencing operation
krb5kdc: starting...
W20260810 19:25:59.062503 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:25:59 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:25:59.284569 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44837
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:59.605451 25066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:59.605736 25066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:59.605851 25066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:59.612998 25066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:59.613116 25066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:59.613183 25066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:59.613267 25066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:59.613325 25066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:59.622949 25066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:44837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:59.624264 25066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:59.625561 25066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:59.637142 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:59.637050 25075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:59.637022 25081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:59.637952 25066 server_base.cc:1034] running on GCE node
I20260810 19:25:59.638705 25066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:59.640094 25066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:59.641292 25066 hybrid_clock.cc:648] HybridClock initialized: now 1786389959641236 us; error 45 us; skew 500 ppm
I20260810 19:25:59.641628 25066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:59 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:59.644584 25066 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:25:59.646104 25066 webserver.cc:460] Webserver started at http://127.19.68.62:45201/ using document root <none> and password file <none>
I20260810 19:25:59.646749 25066 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:59.646876 25066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:59.647176 25066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:59.650214 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "9a139913a27a41b0a90214e3bfdfcc3d"
format_stamp: "Formatted at 2026-08-10 19:25:59 on dist-test-slave-2j7r"
server_key: "048428d48484b36e492002f7f2ec4d55"
server_key_iv: "218e8735ec57e615f27089138c917d18"
server_key_version: "encryptionkey@0"
I20260810 19:25:59.650966 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "9a139913a27a41b0a90214e3bfdfcc3d"
format_stamp: "Formatted at 2026-08-10 19:25:59 on dist-test-slave-2j7r"
server_key: "048428d48484b36e492002f7f2ec4d55"
server_key_iv: "218e8735ec57e615f27089138c917d18"
server_key_version: "encryptionkey@0"
I20260810 19:25:59.656155 25066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:59.659420 25091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:59.660626 25066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:25:59.660773 25066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "9a139913a27a41b0a90214e3bfdfcc3d"
format_stamp: "Formatted at 2026-08-10 19:25:59 on dist-test-slave-2j7r"
server_key: "048428d48484b36e492002f7f2ec4d55"
server_key_iv: "218e8735ec57e615f27089138c917d18"
server_key_version: "encryptionkey@0"
I20260810 19:25:59.660960 25066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:59.673599 25066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:59.677755 25066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:59.678042 25066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:59.695329 25066 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:44837
I20260810 19:25:59.695362 25185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:44837 every 8 connection(s)
I20260810 19:25:59.696722 25066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:59.700441 25186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:59.706276 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25066
I20260810 19:25:59.706476 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:25:59.707103 19728 external_mini_cluster.cc:1428] Setting key 2eae02feaeae9944630a28ddd8c6677f
I20260810 19:25:59.712004 25186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d: Bootstrap starting.
I20260810 19:25:59.715786 25186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:59.717126 25186 log.cc:824] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:59.720693 25186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d: No bootstrap required, opened a new log
Aug 10 19:25:59 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:25:59.731607 25186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } }
I20260810 19:25:59.732092 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:59.709057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.19.68.62:44837)
0810 19:25:59.709897 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:59.709908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:59.709951 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:25:59.711568 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:59.711590 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:59.711959 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:59.712280 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:59.714194 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:59.715438 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:59.717250 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:59.718151 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:59.718355 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:59.723003 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:59.723015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:59.723034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:59.723090 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:59.727292 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:59.727977 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:59.727987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:59.727995 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:59.728240 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:59.728610 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:59.728616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:59.728619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:59.729467 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:59.729842 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:59.730179 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 19:25:59.730656 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":382,"threads_started":1}
I20260810 19:25:59.732115 25186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:59.732254 25186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a139913a27a41b0a90214e3bfdfcc3d, State: Initialized, Role: FOLLOWER
I20260810 19:25:59.732851 25186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } }
I20260810 19:25:59.733107 25186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:59.733280 25186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:59.733465 25186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:59.735900 25186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } }
I20260810 19:25:59.736398 25186 leader_election.cc:302] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a139913a27a41b0a90214e3bfdfcc3d; no voters:
I20260810 19:25:59.736984 25186 leader_election.cc:288] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:59.737079 25193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:59.737785 25193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [term 1 LEADER]: Becoming Leader. State: Replica: 9a139913a27a41b0a90214e3bfdfcc3d, State: Running, Role: LEADER
I20260810 19:25:59.738384 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } }
I20260810 19:25:59.739181 25186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:59.741367 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } } }
I20260810 19:25:59.741461 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a139913a27a41b0a90214e3bfdfcc3d. Latest consensus state: current_term: 1 leader_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a139913a27a41b0a90214e3bfdfcc3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44837 } } }
I20260810 19:25:59.741782 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:59.741782 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:59.743211 25205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:59.748502 25205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:59.757592 25205 catalog_manager.cc:1347] Generated new cluster ID: 3fa5fd26b6c44e60bc35906b0f0aee23
I20260810 19:25:59.757706 25205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:59.776552 25205 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:59.777935 25205 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:59.786823 25205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a139913a27a41b0a90214e3bfdfcc3d: Generated new TSK 0
I20260810 19:25:59.787562 25205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:25:59.845217 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:00.170981 25224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:00.171258 25224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:00.171378 25224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:00.178731 25224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:00.178834 25224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:00.178982 25224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:00.188527 25224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:00.189790 25224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:00.191049 25224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:00.203601 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:00.203692 25234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:00.203729 25232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:00.204458 25224 server_base.cc:1034] running on GCE node
I20260810 19:26:00.205075 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:00.206075 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:00.207276 25224 hybrid_clock.cc:648] HybridClock initialized: now 1786389960207203 us; error 72 us; skew 500 ppm
I20260810 19:26:00.207576 25224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:00 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389960, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:00.210727 25224 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:26:00.212157 25224 webserver.cc:460] Webserver started at http://127.19.68.1:46327/ using document root <none> and password file <none>
I20260810 19:26:00.212808 25224 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:00.212922 25224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:00.213230 25224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:00.216092 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "bf6d797899ea4b1caf027a56404d94da"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "7e67c1002939b20a7fc8e57075b2d4f4"
server_key_iv: "0f222e2278d261a35116eeb4369923ee"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.216872 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "bf6d797899ea4b1caf027a56404d94da"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "7e67c1002939b20a7fc8e57075b2d4f4"
server_key_iv: "0f222e2278d261a35116eeb4369923ee"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.222126 25224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 19:26:00.225494 25247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.226891 25224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:26:00.227041 25224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "bf6d797899ea4b1caf027a56404d94da"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "7e67c1002939b20a7fc8e57075b2d4f4"
server_key_iv: "0f222e2278d261a35116eeb4369923ee"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.227219 25224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:00.240808 25224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:00.244742 25224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:00.245015 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:00.245975 25224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:00.247277 25224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:00.247350 25224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.247468 25224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:00.247532 25224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.277236 25224 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:40125
I20260810 19:26:00.277321 25425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:40125 every 8 connection(s)
I20260810 19:26:00.278535 25224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:00.287952 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25224
I20260810 19:26:00.288136 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:00.288945 19728 external_mini_cluster.cc:1428] Setting key 544deb2a0313982055e2cf5a5f98fede
Aug 10 19:26:00 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389960, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:00.298475 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:00.280947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:46465 (local address 127.19.68.62:44837)
0810 19:26:00.281417 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:00.281421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:00.282330 (+ 909us) server_negotiation.cc:407] Connection header received
0810 19:26:00.284087 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:00.284092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:00.284161 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:00.284251 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:00.285687 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.286499 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.288250 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.288941 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.289074 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:00.293144 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:00.293150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:00.293153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:00.293185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:00.295523 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:00.296338 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:00.296342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:00.296345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:00.296461 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:00.296895 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:00.296898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:00.296900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:00.297190 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:00.297327 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:00.298041 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 19:26:00.298205 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":234,"threads_started":1}
I20260810 19:26:00.299629 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:00.281483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44837 (local address 127.19.68.1:46465)
0810 19:26:00.282167 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:00.282214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:00.283844 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:00.284426 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:00.284434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:00.284938 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:00.285535 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:00.285552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.286674 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.286678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:00.288014 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:00.288028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.289223 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.289229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:00.289426 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:00.290124 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:00.290151 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:00.292886 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:00.295663 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:00.295671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:00.295686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:00.296220 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:00.296563 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:00.296568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:00.296571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:00.296803 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:00.297299 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:00.297306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:00.297678 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 19:26:00.297951 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 19:26:00.298416 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":279,"threads_started":1}
I20260810 19:26:00.301066 25427 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44837
I20260810 19:26:00.301379 25427 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:00.302048 25427 heartbeater.cc:511] Master 127.19.68.62:44837 requested a full tablet report, sending...
I20260810 19:26:00.304136 25126 ts_manager.cc:194] Registered new tserver with Master: bf6d797899ea4b1caf027a56404d94da (127.19.68.1:40125)
I20260810 19:26:00.306082 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:46465
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:26:00.337793 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:00.653335 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:00.653604 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:00.653704 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:00.660679 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:00.660779 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:00.660898 25441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:00.671211 25441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:00.672523 25441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:00.673898 25441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:00.687088 25452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:00.687117 25454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:00.687558 25451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:00.687908 25441 server_base.cc:1034] running on GCE node
I20260810 19:26:00.688452 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:00.689386 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:00.690600 25441 hybrid_clock.cc:648] HybridClock initialized: now 1786389960690563 us; error 27 us; skew 500 ppm
I20260810 19:26:00.690879 25441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:00 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389960, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:00.693857 25441 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:26:00.695426 25441 webserver.cc:460] Webserver started at http://127.19.68.2:41223/ using document root <none> and password file <none>
I20260810 19:26:00.696240 25441 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:00.696390 25441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:00.696717 25441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:00.700438 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "686badc5a9164daebb01d6e604e00d96"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "4d480582506dcb8b404b0f4da3bee4fd"
server_key_iv: "41f6e0fc5a404dcb7fc4f8bd8c6d8b48"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.701440 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "686badc5a9164daebb01d6e604e00d96"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "4d480582506dcb8b404b0f4da3bee4fd"
server_key_iv: "41f6e0fc5a404dcb7fc4f8bd8c6d8b48"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.707733 25441 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:26:00.711889 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.713335 25441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:26:00.713490 25441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "686badc5a9164daebb01d6e604e00d96"
format_stamp: "Formatted at 2026-08-10 19:26:00 on dist-test-slave-2j7r"
server_key: "4d480582506dcb8b404b0f4da3bee4fd"
server_key_iv: "41f6e0fc5a404dcb7fc4f8bd8c6d8b48"
server_key_version: "encryptionkey@0"
I20260810 19:26:00.713717 25441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:00.729450 25441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:00.733500 25441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:00.733745 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:00.734628 25441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:00.735872 25441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:00.735945 25441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.736038 25441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:00.736081 25441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:00.765928 25441 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:45519
I20260810 19:26:00.765993 25653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:45519 every 8 connection(s)
I20260810 19:26:00.767184 25441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:00.776213 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25441
I20260810 19:26:00.776394 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:00.777030 19728 external_mini_cluster.cc:1428] Setting key 67622fa87a47e1a16a6125678994ced7
Aug 10 19:26:00 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389960, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:00.786310 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:00.769575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:38469 (local address 127.19.68.62:44837)
0810 19:26:00.769734 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:00.769738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:00.770859 (+ 1121us) server_negotiation.cc:407] Connection header received
0810 19:26:00.772531 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:00.772535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:00.772605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:00.772687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:00.774112 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.774822 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.776317 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.776952 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.777115 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:00.780619 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:00.780626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:00.780631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:00.780669 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:00.783534 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:00.784174 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:00.784179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:00.784182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:00.784346 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:00.784709 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:00.784713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:00.784715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:00.785023 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:00.785192 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:00.785763 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 19:26:00.785995 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:26:00.787156 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:00.769978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44837 (local address 127.19.68.2:38469)
0810 19:26:00.770701 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:00.770741 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:00.772292 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:00.772866 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:00.772874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:00.773426 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:00.773961 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:00.773979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.774948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.774952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:00.776167 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:00.776180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:00.777200 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:00.777206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:00.777432 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:00.778035 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:00.778056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:00.780410 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:00.783654 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:00.783660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:00.783671 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:00.784066 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:00.784440 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:00.784443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:00.784445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:00.784617 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:00.785153 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:00.785158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:00.785440 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 19:26:00.785658 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:26:00.786044 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":295,"threads_started":1}
I20260810 19:26:00.788525 25654 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44837
I20260810 19:26:00.788784 25654 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:00.789505 25654 heartbeater.cc:511] Master 127.19.68.62:44837 requested a full tablet report, sending...
I20260810 19:26:00.790954 25126 ts_manager.cc:194] Registered new tserver with Master: 686badc5a9164daebb01d6e604e00d96 (127.19.68.2:45519)
I20260810 19:26:00.792215 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:38469
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:26:00.827956 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:01.146868 25662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:01.147132 25662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:01.147267 25662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:01.154274 25662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:01.154371 25662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:01.154518 25662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:01.163856 25662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:01.165189 25662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:01.166528 25662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:01.179239 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:01.179198 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:01.179280 25672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:01.180482 25662 server_base.cc:1034] running on GCE node
I20260810 19:26:01.181036 25662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:01.181939 25662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:01.183139 25662 hybrid_clock.cc:648] HybridClock initialized: now 1786389961183097 us; error 34 us; skew 500 ppm
I20260810 19:26:01.183422 25662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:01.186471 25662 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:26:01.187819 25662 webserver.cc:460] Webserver started at http://127.19.68.3:44747/ using document root <none> and password file <none>
I20260810 19:26:01.188438 25662 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:01.188553 25662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:01.188848 25662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:01.191642 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "3d5d95be674343d0ad1c8619a0c347d7"
format_stamp: "Formatted at 2026-08-10 19:26:01 on dist-test-slave-2j7r"
server_key: "91dabfa9451bb2692a37434d4e65a90b"
server_key_iv: "6560610d1d943fa0d36a82d487e1d3f3"
server_key_version: "encryptionkey@0"
I20260810 19:26:01.192384 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "3d5d95be674343d0ad1c8619a0c347d7"
format_stamp: "Formatted at 2026-08-10 19:26:01 on dist-test-slave-2j7r"
server_key: "91dabfa9451bb2692a37434d4e65a90b"
server_key_iv: "6560610d1d943fa0d36a82d487e1d3f3"
server_key_version: "encryptionkey@0"
I20260810 19:26:01.197224 25662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260810 19:26:01.200555 25686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:01.201903 25662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:01.202066 25662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "3d5d95be674343d0ad1c8619a0c347d7"
format_stamp: "Formatted at 2026-08-10 19:26:01 on dist-test-slave-2j7r"
server_key: "91dabfa9451bb2692a37434d4e65a90b"
server_key_iv: "6560610d1d943fa0d36a82d487e1d3f3"
server_key_version: "encryptionkey@0"
I20260810 19:26:01.202250 25662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:01.216387 25662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:01.220501 25662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:01.220789 25662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:01.221745 25662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:01.223083 25662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:01.223161 25662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:01.223284 25662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:01.223344 25662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:01.254253 25662 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:35005
I20260810 19:26:01.254297 25867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:35005 every 8 connection(s)
I20260810 19:26:01.255504 25662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:01.257050 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25662
I20260810 19:26:01.257251 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:01.257824 19728 external_mini_cluster.cc:1428] Setting key bbf095836f319843001d6967644f8321
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:01.274991 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.258260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:38767 (local address 127.19.68.62:44837)
0810 19:26:01.258475 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:01.258482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:01.259742 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 19:26:01.261743 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:01.261747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:01.261833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:01.261920 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:01.263380 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.264126 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.265850 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.266281 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.266382 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.269694 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:01.269699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:01.269702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:01.269751 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:01.272303 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.273022 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.273027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.273030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.273183 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.273603 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.273606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.273608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.273885 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:01.273992 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:01.274570 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 19:26:01.274742 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:26:01.275894 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.258798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44837 (local address 127.19.68.3:38767)
0810 19:26:01.259589 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:01.259630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:01.261490 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:01.262101 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:01.262109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:01.262639 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:01.263222 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:01.263248 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.264297 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.264301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:01.265707 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:01.265720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.266510 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.266515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:01.266715 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.267305 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:01.267327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:01.269514 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:01.272429 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:01.272435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:01.272446 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:01.272906 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:01.273288 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:01.273291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:01.273294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:01.273505 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:01.273973 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:01.273978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:01.274261 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 19:26:01.274486 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:26:01.274845 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":341,"threads_started":1}
I20260810 19:26:01.277197 25869 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44837
I20260810 19:26:01.277534 25869 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:01.278420 25869 heartbeater.cc:511] Master 127.19.68.62:44837 requested a full tablet report, sending...
I20260810 19:26:01.279935 25126 ts_manager.cc:194] Registered new tserver with Master: 3d5d95be674343d0ad1c8619a0c347d7 (127.19.68.3:35005)
I20260810 19:26:01.280789 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:38767
I20260810 19:26:01.285218 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:26:01.308972 25427 heartbeater.cc:503] Master 127.19.68.62:44837 was elected leader, sending a full tablet report...
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:01.337415 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.323914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.19.68.62:44837)
0810 19:26:01.324079 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:01.324085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:01.324576 (+ 491us) server_negotiation.cc:407] Connection header received
0810 19:26:01.324766 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:01.324771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:01.324837 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:01.324920 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:01.326364 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.327088 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.328333 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.328822 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.328936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.332620 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:01.332626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:01.332629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:01.332662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:01.334948 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.335548 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.335554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.335556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.335682 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.336037 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.336041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.336043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.336350 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:01.336574 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:01.336867 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 19:26:01.337107 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 19:26:01.337952 25126 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38782
I20260810 19:26:01.340171 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25224
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:01.355051 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.341120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.19.68.1:40125)
0810 19:26:01.341688 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:01.341695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:01.341722 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:26:01.341783 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:01.341789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:01.341998 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:01.342129 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:01.343277 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.344412 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.345710 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.346287 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.346388 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.349993 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:01.350004 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:01.350025 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:01.350064 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:01.352639 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.353327 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.353336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.353339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.353471 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.353784 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.353787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.353789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.354125 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:01.354337 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:01.354548 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 19:26:01.354741 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
W20260810 19:26:01.356355 25421 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52728
W20260810 19:26:01.357465 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25224: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:26:01.357687 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25224
I20260810 19:26:01.382608 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25441
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:26:01.398795 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.383536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.19.68.2:45519)
0810 19:26:01.384617 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:01.384625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:01.384650 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:26:01.384714 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:01.384721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:01.384930 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:01.385079 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:01.386384 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.387379 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.388602 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.389243 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.389356 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.393269 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:01.393282 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:01.393295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:01.393327 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:01.396006 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.396637 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.396645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.396647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.396782 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.397175 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.397180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.397184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.397531 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:01.397840 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:01.398175 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 19:26:01.398454 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":816,"threads_started":1}
W20260810 19:26:01.400056 25647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44224
W20260810 19:26:01.401162 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25441: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:26:01.401335 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25441
I20260810 19:26:01.427189 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25662
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:26:01.443771 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:01.428119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.19.68.3:35005)
0810 19:26:01.428728 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:01.428736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:01.428768 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:26:01.428837 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:01.428843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:01.429055 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:01.429231 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:01.430609 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.431630 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.432852 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:01.433494 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:01.433612 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:01.438144 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:01.438158 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:01.438173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:01.438210 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:01.440909 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.441635 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.441643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.441645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.441782 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:01.442133 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:01.442137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:01.442139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:01.442496 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:01.442815 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:01.443092 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 19:26:01.443370 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":266,"threads_started":1}
W20260810 19:26:01.445119 25861 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36820
W20260810 19:26:01.446230 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:26:01.446394 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25662
I20260810 19:26:01.465306 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25066
I20260810 19:26:01.466441 25178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:01.570052 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25066
2026-08-10T19:26:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3740 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25892](info): set up 2 sockets
Aug 10 19:26:01 dist-test-slave-2j7r krb5kdc[25892](info): commencing operation
krb5kdc: starting...
W20260810 19:26:02.806692 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:02 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:26:03.029749 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35303
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:03.348510 25914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:03.348769 25914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:03.348897 25914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:03.355921 25914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:03.356011 25914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:03.356082 25914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:03.356159 25914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:03.356215 25914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:03.365676 25914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:03.366936 25914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:03.368175 25914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:03.379457 25925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:03.379522 25929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:03.379567 25926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:03.380620 25914 server_base.cc:1034] running on GCE node
I20260810 19:26:03.381409 25914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:03.382692 25914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:03.383899 25914 hybrid_clock.cc:648] HybridClock initialized: now 1786389963383856 us; error 32 us; skew 500 ppm
I20260810 19:26:03.384224 25914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:03 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389963, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:03.391950 25914 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:26:03.393466 25914 webserver.cc:460] Webserver started at http://127.19.68.62:43191/ using document root <none> and password file <none>
I20260810 19:26:03.394109 25914 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:03.394217 25914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:03.394536 25914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:03.397514 25914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "3b4241b6c0fe447dab5b3c13741321be"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "303bfb6cee9ad9b48ff330b1ee6d6692"
server_key_iv: "bf459db02633065de87d4e4509cb338d"
server_key_version: "encryptionkey@0"
I20260810 19:26:03.398285 25914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "3b4241b6c0fe447dab5b3c13741321be"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "303bfb6cee9ad9b48ff330b1ee6d6692"
server_key_iv: "bf459db02633065de87d4e4509cb338d"
server_key_version: "encryptionkey@0"
I20260810 19:26:03.403257 25914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 19:26:03.406577 25936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:03.407802 25914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:26:03.407953 25914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "3b4241b6c0fe447dab5b3c13741321be"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "303bfb6cee9ad9b48ff330b1ee6d6692"
server_key_iv: "bf459db02633065de87d4e4509cb338d"
server_key_version: "encryptionkey@0"
I20260810 19:26:03.408118 25914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:03.431037 25914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:03.434974 25914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:03.435256 25914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:03.452664 25914 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:35303
I20260810 19:26:03.452694 26026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:35303 every 8 connection(s)
I20260810 19:26:03.453997 25914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:03.457547 26029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:03.459656 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25914
I20260810 19:26:03.459880 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:03.460495 19728 external_mini_cluster.cc:1428] Setting key 1a11d146c4b0f39ea5d91a9bc4474cb8
I20260810 19:26:03.468823 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: Bootstrap starting.
I20260810 19:26:03.472438 26029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:03.473732 26029 log.cc:824] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:26:03 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:03.476491 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: No bootstrap required, opened a new log
I20260810 19:26:03.482828 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:03.462465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.19.68.62:35303)
0810 19:26:03.463223 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:03.463239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:03.463283 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:26:03.464941 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:03.464967 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:03.465396 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:03.465706 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:03.467515 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:03.468790 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:03.470496 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:03.471164 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:03.471295 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:03.475922 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:03.475930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:03.475943 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:03.475982 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:03.478936 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:03.479576 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:03.479582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:03.479591 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:03.479754 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:03.480132 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:03.480136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:03.480138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:03.480683 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:03.481001 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:03.481459 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 19:26:03.481872 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":351,"threads_started":1}
I20260810 19:26:03.484810 26029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } }
I20260810 19:26:03.485194 26029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:03.485327 26029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4241b6c0fe447dab5b3c13741321be, State: Initialized, Role: FOLLOWER
I20260810 19:26:03.485927 26029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } }
I20260810 19:26:03.486182 26029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:03.486299 26029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:03.486429 26029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:03.488835 26029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } }
I20260810 19:26:03.489390 26029 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4241b6c0fe447dab5b3c13741321be; no voters:
I20260810 19:26:03.491812 26029 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:03.491941 26036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:03.492620 26036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [term 1 LEADER]: Becoming Leader. State: Replica: 3b4241b6c0fe447dab5b3c13741321be, State: Running, Role: LEADER
I20260810 19:26:03.493300 26036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } }
I20260810 19:26:03.494299 26029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:03.496297 26037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4241b6c0fe447dab5b3c13741321be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } } }
I20260810 19:26:03.496995 26037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:03.497311 26038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4241b6c0fe447dab5b3c13741321be. Latest consensus state: current_term: 1 leader_uuid: "3b4241b6c0fe447dab5b3c13741321be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4241b6c0fe447dab5b3c13741321be" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35303 } } }
I20260810 19:26:03.497645 26038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be [sys.catalog]: This master's current role is: LEADER
W20260810 19:26:03.507270 26059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:26:03.507413 26059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:26:03.509584 26047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:03.511217 26047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:03.519552 26047 catalog_manager.cc:1347] Generated new cluster ID: 89b9d5c99d4a46b9872f003938455545
I20260810 19:26:03.519642 26047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:03.538056 26047 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:03.539429 26047 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:03.546152 26047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4241b6c0fe447dab5b3c13741321be: Generated new TSK 0
I20260810 19:26:03.547134 26047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:26:03.610662 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:03.945209 26070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:03.945533 26070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:03.945689 26070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:03.953375 26070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:03.953514 26070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:03.953676 26070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:03.963845 26070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:03.965227 26070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:03.966660 26070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:03.979933 26077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:03.979892 26079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:03.980330 26082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:03.980639 26070 server_base.cc:1034] running on GCE node
I20260810 19:26:03.981282 26070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:03.982249 26070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:03.983462 26070 hybrid_clock.cc:648] HybridClock initialized: now 1786389963983417 us; error 38 us; skew 500 ppm
I20260810 19:26:03.983767 26070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:03 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389963, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:03.986835 26070 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:26:03.988153 26070 webserver.cc:460] Webserver started at http://127.19.68.1:35541/ using document root <none> and password file <none>
I20260810 19:26:03.988785 26070 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:03.988900 26070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:03.989230 26070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:03.992151 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "fab5b3ba686c45cebb532d9205eafbbe"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "0e3fe55623256982379c79fca6801718"
server_key_iv: "5192ed544e861de7c65a67895000e378"
server_key_version: "encryptionkey@0"
I20260810 19:26:03.992924 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "fab5b3ba686c45cebb532d9205eafbbe"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "0e3fe55623256982379c79fca6801718"
server_key_iv: "5192ed544e861de7c65a67895000e378"
server_key_version: "encryptionkey@0"
I20260810 19:26:03.998111 26070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:26:04.001681 26089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.003002 26070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:26:04.003140 26070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "fab5b3ba686c45cebb532d9205eafbbe"
format_stamp: "Formatted at 2026-08-10 19:26:03 on dist-test-slave-2j7r"
server_key: "0e3fe55623256982379c79fca6801718"
server_key_iv: "5192ed544e861de7c65a67895000e378"
server_key_version: "encryptionkey@0"
I20260810 19:26:04.003319 26070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:04.040576 26070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:04.044641 26070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:04.044911 26070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:04.046037 26070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:04.047356 26070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:04.047443 26070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.047559 26070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:04.047612 26070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.078296 26070 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:43543
I20260810 19:26:04.078367 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:43543 every 8 connection(s)
I20260810 19:26:04.079533 26070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:04.083642 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26070
I20260810 19:26:04.083874 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:04.084487 19728 external_mini_cluster.cc:1428] Setting key 2415cf7c090f43a81db653d68caa3d32
Aug 10 19:26:04 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389963, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:04.100967 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:04.082053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:36501 (local address 127.19.68.62:35303)
0810 19:26:04.082515 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:04.082519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:04.083440 (+ 921us) server_negotiation.cc:407] Connection header received
0810 19:26:04.085535 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:04.085539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:04.085606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:04.085696 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:04.087628 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.088326 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.090425 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.090831 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.090927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:04.094781 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:04.094786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:04.094789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:04.094820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:04.097877 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:04.098585 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:04.098590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:04.098594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:04.098739 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:04.099176 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:04.099181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:04.099184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:04.099567 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:04.099709 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:04.100388 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 19:26:04.100611 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260810 19:26:04.101776 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:04.082581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35303 (local address 127.19.68.1:36501)
0810 19:26:04.083281 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:04.083320 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:04.085105 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:04.086005 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:04.086016 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:04.086659 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:04.087443 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:04.087473 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.088486 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.088491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:04.090273 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:04.090289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.091051 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.091059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:04.091318 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:04.091960 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:04.091987 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:04.094554 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:04.098000 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:04.098006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:04.098017 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:04.098455 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:04.098848 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:04.098852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:04.098854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:04.099065 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:04.099679 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:04.099685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:04.100009 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 19:26:04.100232 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 19:26:04.100620 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
I20260810 19:26:04.103252 26271 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35303
I20260810 19:26:04.103565 26271 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:04.104255 26271 heartbeater.cc:511] Master 127.19.68.62:35303 requested a full tablet report, sending...
I20260810 19:26:04.106433 25968 ts_manager.cc:194] Registered new tserver with Master: fab5b3ba686c45cebb532d9205eafbbe (127.19.68.1:43543)
I20260810 19:26:04.107978 25968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:36501
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:26:04.135329 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:04.452613 26283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:04.452898 26283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:04.453034 26283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:04.460160 26283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:04.460274 26283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:04.460433 26283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:04.470062 26283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:04.471356 26283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:04.472666 26283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:04.487555 26293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:04.487763 26291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:04.488010 26288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:04.488867 26283 server_base.cc:1034] running on GCE node
I20260810 19:26:04.489524 26283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:04.490507 26283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:04.491716 26283 hybrid_clock.cc:648] HybridClock initialized: now 1786389964491650 us; error 59 us; skew 500 ppm
I20260810 19:26:04.492026 26283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:04 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:04.495134 26283 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:26:04.496549 26283 webserver.cc:460] Webserver started at http://127.19.68.2:44971/ using document root <none> and password file <none>
I20260810 19:26:04.497224 26283 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:04.497336 26283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:04.497601 26283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:04.500545 26283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "3f9874d70b6c4fa38e705d46cebd2610"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "30754d7510280f1d2c7f66d81c7317af"
server_key_iv: "e2694f309f2bac25eabd3fd1d1cfecab"
server_key_version: "encryptionkey@0"
I20260810 19:26:04.501474 26283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "3f9874d70b6c4fa38e705d46cebd2610"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "30754d7510280f1d2c7f66d81c7317af"
server_key_iv: "e2694f309f2bac25eabd3fd1d1cfecab"
server_key_version: "encryptionkey@0"
I20260810 19:26:04.507838 26283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:26:04.511201 26308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.512533 26283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:04.512693 26283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "3f9874d70b6c4fa38e705d46cebd2610"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "30754d7510280f1d2c7f66d81c7317af"
server_key_iv: "e2694f309f2bac25eabd3fd1d1cfecab"
server_key_version: "encryptionkey@0"
I20260810 19:26:04.512883 26283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:04.534055 26283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:04.538069 26283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:04.538317 26283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:04.539225 26283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:04.540473 26283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:04.540548 26283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.540632 26283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:04.540675 26283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:04.570876 26283 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:46363
I20260810 19:26:04.570945 26503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:46363 every 8 connection(s)
I20260810 19:26:04.572124 26283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:04.575358 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26283
I20260810 19:26:04.575567 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:04.576159 19728 external_mini_cluster.cc:1428] Setting key 1a5f675f3a02253706554cf236593d85
Aug 10 19:26:04 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:04.595664 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:04.574749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:54683 (local address 127.19.68.62:35303)
0810 19:26:04.574945 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:04.574950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:04.576338 (+ 1388us) server_negotiation.cc:407] Connection header received
0810 19:26:04.578957 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:04.578961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:04.579029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:04.579115 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:04.581189 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.581901 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.584066 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.584506 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.584599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:04.588655 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:04.588661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:04.588666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:04.588707 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:04.592025 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:04.592800 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:04.592806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:04.592810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:04.592984 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:04.593523 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:04.593529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:04.593533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:04.593959 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:04.594112 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:04.594912 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 19:26:04.595207 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:26:04.596355 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:04.575216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35303 (local address 127.19.68.2:54683)
0810 19:26:04.576100 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:04.576153 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:04.578627 (+ 2474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:04.579413 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:04.579425 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:04.580133 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:04.580981 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:04.581010 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.582066 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.582071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:04.583897 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:04.583914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:04.584739 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:04.584748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:04.585054 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:04.585821 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:04.585852 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:04.588416 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:04.592177 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:04.592184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:04.592196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:04.592648 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:04.593151 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:04.593154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:04.593157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:04.593378 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:04.594096 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:04.594102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:04.594447 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 19:26:04.594726 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 19:26:04.595153 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":324,"threads_started":1}
I20260810 19:26:04.597973 26505 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35303
I20260810 19:26:04.598280 26505 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:04.599040 26505 heartbeater.cc:511] Master 127.19.68.62:35303 requested a full tablet report, sending...
I20260810 19:26:04.600600 25968 ts_manager.cc:194] Registered new tserver with Master: 3f9874d70b6c4fa38e705d46cebd2610 (127.19.68.2:46363)
I20260810 19:26:04.601459 25968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:54683
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:26:04.631837 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:04.956208 26515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:04.956486 26515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:04.956588 26515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:04.963913 26515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:04.964033 26515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:04.964166 26515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:04.974232 26515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:04.975486 26515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:04.976985 26515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:04.990198 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:04.990150 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:04.990267 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:04.991364 26515 server_base.cc:1034] running on GCE node
I20260810 19:26:04.991947 26515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:04.992791 26515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:04.994047 26515 hybrid_clock.cc:648] HybridClock initialized: now 1786389964994004 us; error 37 us; skew 500 ppm
I20260810 19:26:04.994315 26515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:04 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:04.997538 26515 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:26:04.998950 26515 webserver.cc:460] Webserver started at http://127.19.68.3:46551/ using document root <none> and password file <none>
I20260810 19:26:04.999596 26515 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:04.999711 26515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:05.000020 26515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:05.002874 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "136616fc73b34310b5bd06dd339c3440"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "72c93ca46e6a4ca0c069384ac33d425a"
server_key_iv: "d7c7c5ae33d81bd7de471c0a431c6cde"
server_key_version: "encryptionkey@0"
I20260810 19:26:05.003690 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "136616fc73b34310b5bd06dd339c3440"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "72c93ca46e6a4ca0c069384ac33d425a"
server_key_iv: "d7c7c5ae33d81bd7de471c0a431c6cde"
server_key_version: "encryptionkey@0"
I20260810 19:26:05.008955 26515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:26:05.012475 26539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:05.013903 26515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:05.014057 26515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "136616fc73b34310b5bd06dd339c3440"
format_stamp: "Formatted at 2026-08-10 19:26:04 on dist-test-slave-2j7r"
server_key: "72c93ca46e6a4ca0c069384ac33d425a"
server_key_iv: "d7c7c5ae33d81bd7de471c0a431c6cde"
server_key_version: "encryptionkey@0"
I20260810 19:26:05.014240 26515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:05.033701 26515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:05.037575 26515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:05.037868 26515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:05.038798 26515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:05.040082 26515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:05.040175 26515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:05.040295 26515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:05.040356 26515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:05.069983 26515 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:46679
I20260810 19:26:05.070026 26730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:46679 every 8 connection(s)
I20260810 19:26:05.071218 26515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:05.074518 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26515
I20260810 19:26:05.074734 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:05.075301 19728 external_mini_cluster.cc:1428] Setting key 58e3168e4440668aea431260e9176870
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:05.091747 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.073828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:48715 (local address 127.19.68.62:35303)
0810 19:26:05.074020 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.074025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.075382 (+ 1357us) server_negotiation.cc:407] Connection header received
0810 19:26:05.077536 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.077542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.077658 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.077772 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.079466 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.080594 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.082277 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.082892 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.083023 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.086276 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.086281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.086284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.086318 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.088826 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.089667 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.089672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.089676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.089853 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.090288 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.090291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.090293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.090602 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.090738 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.091316 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.091497 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:26:05.092559 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.074364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35303 (local address 127.19.68.3:48715)
0810 19:26:05.075215 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:05.075264 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:05.077253 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:05.078135 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:05.078146 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:05.078743 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:05.079286 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:05.079303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.080756 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.080760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:05.082134 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:05.082146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.083146 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.083151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:05.083352 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.083907 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:05.083932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:05.086087 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:05.088967 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:05.088973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:05.088985 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:05.089575 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:05.089962 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:05.089967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:05.089970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:05.090195 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:05.090724 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:05.090730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:05.091012 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 19:26:05.091222 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 19:26:05.091580 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":379,"threads_started":1}
I20260810 19:26:05.093735 26732 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35303
I20260810 19:26:05.094000 26732 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:05.094705 26732 heartbeater.cc:511] Master 127.19.68.62:35303 requested a full tablet report, sending...
I20260810 19:26:05.096036 25968 ts_manager.cc:194] Registered new tserver with Master: 136616fc73b34310b5bd06dd339c3440 (127.19.68.3:46679)
I20260810 19:26:05.096856 25968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:48715
I20260810 19:26:05.102843 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:26:05.110733 26271 heartbeater.cc:503] Master 127.19.68.62:35303 was elected leader, sending a full tablet report...
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:05.156248 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.142436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.19.68.1:43543)
0810 19:26:05.143042 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.143050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.143077 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:26:05.143150 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.143157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.143358 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.143488 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.144779 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.145848 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.147048 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.147600 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.147693 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.151307 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.151319 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.151332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.151365 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.153972 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.154540 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.154546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.154549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.154675 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.154964 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.154967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.154969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.155282 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.155516 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.155692 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.155900 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":299,"threads_started":1}
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:05.172986 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.160317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.19.68.62:35303)
0810 19:26:05.160522 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.160526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.160637 (+ 111us) server_negotiation.cc:407] Connection header received
0810 19:26:05.160756 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.160759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.160821 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.160898 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.162142 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.162895 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.164063 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.164495 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.164595 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.168177 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.168182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.168184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.168215 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.170924 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.171448 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.171451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.171454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.171580 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.171838 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.171841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.171843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.172136 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.172305 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.172541 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.172695 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 19:26:05.173616 25968 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596
I20260810 19:26:05.174546 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26070
I20260810 19:26:05.186472 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.175499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.19.68.1:43543)
0810 19:26:05.175687 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.175715 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.175784 (+ 69us) server_negotiation.cc:407] Connection header received
0810 19:26:05.175943 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.175949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.176021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.176102 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.177293 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.178098 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.179262 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.179679 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.179783 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.181861 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.181866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.181869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.181899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.184135 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.184705 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.184711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.184714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.184835 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.185186 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.185191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.185195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.185436 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.185654 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.185978 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.186193 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:05.187036 26260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:05.268057 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26070
I20260810 19:26:05.297010 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26283
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:26:05.313292 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.297909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.19.68.2:46363)
0810 19:26:05.298478 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.298484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.298512 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:26:05.298603 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.298611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.298854 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.298979 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.300221 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.301355 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.302664 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.303206 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.303304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.307759 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.307771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.307784 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.307827 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.310512 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.311223 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.311231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.311233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.311364 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.311664 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.311669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.311671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.312056 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.312349 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.312568 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.312800 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":276,"threads_started":1}
I20260810 19:26:05.314492 26498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:05.396514 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26283
I20260810 19:26:05.423112 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26515
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:26:05.438933 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:05.424029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.19.68.3:46679)
0810 19:26:05.424703 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:05.424709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:05.424750 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:26:05.424811 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:05.424817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:05.425022 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:05.425179 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:05.426480 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.427528 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.428761 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:05.429554 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:05.429650 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:05.433761 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:05.433787 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:05.433800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:05.433838 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:05.436545 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.437129 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.437137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.437139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.437265 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:05.437576 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:05.437578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:05.437580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:05.437931 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:05.438149 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:05.438367 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 19:26:05.438569 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":345,"threads_started":1}
I20260810 19:26:05.440270 26723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:05.523937 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26515
I20260810 19:26:05.544855 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25914
I20260810 19:26:05.546267 26022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:05.650022 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25914
2026-08-10T19:26:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4086 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[26763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[26763](info): set up 2 sockets
Aug 10 19:26:05 dist-test-slave-2j7r krb5kdc[26763](info): commencing operation
krb5kdc: starting...
W20260810 19:26:06.913691 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:07 dist-test-slave-2j7r krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:26:07.130748 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40087
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:40087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:07.447463 26783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:07.447731 26783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:07.447862 26783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:07.454839 26783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:07.454936 26783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:07.455015 26783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:07.455084 26783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:07.455147 26783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:07.464655 26783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:40087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:07.465938 26783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:07.467233 26783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:07.478518 26797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:07.478551 26794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:07.478471 26792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:07.479434 26783 server_base.cc:1034] running on GCE node
I20260810 19:26:07.480129 26783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:07.481506 26783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:07.482733 26783 hybrid_clock.cc:648] HybridClock initialized: now 1786389967482685 us; error 33 us; skew 500 ppm
I20260810 19:26:07.483090 26783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:07 dist-test-slave-2j7r krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:07.486227 26783 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:26:07.487427 26783 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:07.487520 26783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:07.487854 26783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:07.490816 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "a77f19f05fb14174a379638dd19dfbd4"
format_stamp: "Formatted at 2026-08-10 19:26:07 on dist-test-slave-2j7r"
server_key: "9db78bd6b26b40c31c3406d08553a82b"
server_key_iv: "313f6eac2ce1b5538cd58393350b482d"
server_key_version: "encryptionkey@0"
I20260810 19:26:07.491605 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "a77f19f05fb14174a379638dd19dfbd4"
format_stamp: "Formatted at 2026-08-10 19:26:07 on dist-test-slave-2j7r"
server_key: "9db78bd6b26b40c31c3406d08553a82b"
server_key_iv: "313f6eac2ce1b5538cd58393350b482d"
server_key_version: "encryptionkey@0"
I20260810 19:26:07.496603 26783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:26:07.500108 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:07.501439 26783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:07.501596 26783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "a77f19f05fb14174a379638dd19dfbd4"
format_stamp: "Formatted at 2026-08-10 19:26:07 on dist-test-slave-2j7r"
server_key: "9db78bd6b26b40c31c3406d08553a82b"
server_key_iv: "313f6eac2ce1b5538cd58393350b482d"
server_key_version: "encryptionkey@0"
I20260810 19:26:07.501796 26783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:07.527199 26783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:07.531409 26783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:07.531654 26783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:07.548736 26783 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:40087
I20260810 19:26:07.548758 26894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:40087 every 8 connection(s)
I20260810 19:26:07.549631 26783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:07.551115 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26783
I20260810 19:26:07.551326 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:07.551963 19728 external_mini_cluster.cc:1428] Setting key b79da1fc98416ae9361e2cfaaf798201
I20260810 19:26:07.553620 26895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:07.566525 26895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: Bootstrap starting.
Aug 10 19:26:07 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:07.570191 26895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:07.571512 26895 log.cc:824] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:07.574183 26895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: No bootstrap required, opened a new log
I20260810 19:26:07.576139 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:07.553881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.19.68.62:40087)
0810 19:26:07.554716 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:07.554731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:07.554781 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:26:07.556728 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:07.556755 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:07.557282 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:07.557692 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:07.559078 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:07.561073 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:07.562326 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:07.563300 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:07.563480 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:07.567910 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:07.567925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:07.567945 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:07.567996 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:07.571819 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:07.572583 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:07.572590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:07.572596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:07.572809 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:07.573295 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:07.573301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:07.573303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:07.573882 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:07.574190 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:07.574737 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 19:26:07.575134 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":355,"threads_started":1}
I20260810 19:26:07.582492 26895 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } }
I20260810 19:26:07.582832 26895 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:07.582918 26895 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77f19f05fb14174a379638dd19dfbd4, State: Initialized, Role: FOLLOWER
I20260810 19:26:07.583424 26895 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } }
I20260810 19:26:07.583650 26895 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:07.583771 26895 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:07.583922 26895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:07.586050 26895 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } }
I20260810 19:26:07.586504 26895 leader_election.cc:302] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77f19f05fb14174a379638dd19dfbd4; no voters:
I20260810 19:26:07.587062 26895 leader_election.cc:288] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:07.587204 26904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:07.588218 26904 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [term 1 LEADER]: Becoming Leader. State: Replica: a77f19f05fb14174a379638dd19dfbd4, State: Running, Role: LEADER
I20260810 19:26:07.588764 26904 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } }
I20260810 19:26:07.589243 26895 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:07.591501 26906 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77f19f05fb14174a379638dd19dfbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } } }
I20260810 19:26:07.591843 26906 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:07.592262 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77f19f05fb14174a379638dd19dfbd4. Latest consensus state: current_term: 1 leader_uuid: "a77f19f05fb14174a379638dd19dfbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f19f05fb14174a379638dd19dfbd4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40087 } } }
I20260810 19:26:07.592589 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:07.593039 26915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:07.598762 26915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:07.607321 26915 catalog_manager.cc:1347] Generated new cluster ID: 06101da15b784e959467e431ac87a5ac
I20260810 19:26:07.607419 26915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:07.621055 26915 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:07.622293 26915 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:07.634465 26915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: Generated new TSK 0
I20260810 19:26:07.635319 26915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:26:07.700170 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:08.061674 26935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:08.061954 26935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:08.062083 26935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:08.069262 26935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:08.069365 26935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:08.069514 26935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:08.079245 26935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:08.080500 26935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:08.081805 26935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:08.094164 26942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:08.094748 26944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:08.094796 26948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:08.095439 26935 server_base.cc:1034] running on GCE node
I20260810 19:26:08.095995 26935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:08.096874 26935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:08.098086 26935 hybrid_clock.cc:648] HybridClock initialized: now 1786389968098023 us; error 47 us; skew 500 ppm
I20260810 19:26:08.098423 26935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:08 dist-test-slave-2j7r krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389968, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:08.101504 26935 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:26:08.102633 26935 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:08.102735 26935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:08.103036 26935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:08.106144 26935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "e92c24eecff54e529aa4f3d25efe1988"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "e2a73f0caff6c337cac927fa1da3212a"
server_key_iv: "4cc3d7a7d327bfc3e86f411a540a91fe"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.106959 26935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "e92c24eecff54e529aa4f3d25efe1988"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "e2a73f0caff6c337cac927fa1da3212a"
server_key_iv: "4cc3d7a7d327bfc3e86f411a540a91fe"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.112051 26935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:26:08.115406 26954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.116638 26935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:26:08.116786 26935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "e92c24eecff54e529aa4f3d25efe1988"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "e2a73f0caff6c337cac927fa1da3212a"
server_key_iv: "4cc3d7a7d327bfc3e86f411a540a91fe"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.116982 26935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:08.131716 26935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:08.136067 26935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:08.136315 26935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:08.137281 26935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:08.138592 26935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:08.138689 26935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.138792 26935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:08.138885 26935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.169183 26935 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:44285
I20260810 19:26:08.169323 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:44285 every 8 connection(s)
I20260810 19:26:08.170058 26935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:08.176388 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26935
I20260810 19:26:08.176656 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:08.177639 19728 external_mini_cluster.cc:1428] Setting key c88d152685dce91de0e30dd037890b00
Aug 10 19:26:08 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389968, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:08.191761 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:08.172478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:59249 (local address 127.19.68.62:40087)
0810 19:26:08.172939 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:08.172944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:08.173848 (+ 904us) server_negotiation.cc:407] Connection header received
0810 19:26:08.176189 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:08.176193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:08.176262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:08.176355 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:08.178504 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.179480 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.181845 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.182235 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.182326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:08.186228 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:08.186233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:08.186236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:08.186268 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:08.188686 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:08.189462 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:08.189466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:08.189468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:08.189591 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:08.190032 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:08.190036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:08.190038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:08.190348 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:08.190506 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:08.191184 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 19:26:08.191423 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260810 19:26:08.192615 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:08.172861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40087 (local address 127.19.68.1:59249)
0810 19:26:08.173651 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:08.173698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:08.175852 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:08.176638 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:08.176653 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:08.177448 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:08.178274 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:08.178303 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.179667 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.179673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:08.181686 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:08.181704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.182466 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.182474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:08.182755 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:08.183447 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:08.183487 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:08.186019 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:08.188817 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:08.188826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:08.188841 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:08.189342 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:08.189736 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:08.189739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:08.189742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:08.189919 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:08.190477 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:08.190486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:08.190796 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 19:26:08.191042 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 19:26:08.191526 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":347,"threads_started":1}
I20260810 19:26:08.194145 27142 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40087
I20260810 19:26:08.194484 27142 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:08.195407 27142 heartbeater.cc:511] Master 127.19.68.62:40087 requested a full tablet report, sending...
I20260810 19:26:08.197926 26835 ts_manager.cc:194] Registered new tserver with Master: e92c24eecff54e529aa4f3d25efe1988 (127.19.68.1:44285)
I20260810 19:26:08.200098 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:59249
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:26:08.231842 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:08.553318 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:08.553570 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:08.553709 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:08.560742 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:08.560843 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:08.561007 27158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:08.570533 27158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:08.571795 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:08.573071 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:08.585611 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:08.585644 27174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:08.585681 27168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:08.586457 27158 server_base.cc:1034] running on GCE node
I20260810 19:26:08.586994 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:08.587842 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:08.589053 27158 hybrid_clock.cc:648] HybridClock initialized: now 1786389968589005 us; error 38 us; skew 500 ppm
I20260810 19:26:08.589452 27158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:08 dist-test-slave-2j7r krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389968, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:08.597294 27158 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:26:08.598448 27158 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:08.598551 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:08.598866 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:08.601881 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "aa7ed972452b4962ae82dcf95139e4f2"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "fbdd5f39780b61f848555ef65ce3f59f"
server_key_iv: "aec53b23071250e316473e79440da15a"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.602664 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "aa7ed972452b4962ae82dcf95139e4f2"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "fbdd5f39780b61f848555ef65ce3f59f"
server_key_iv: "aec53b23071250e316473e79440da15a"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.607527 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:08.610900 27188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.612174 27158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:08.612358 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "aa7ed972452b4962ae82dcf95139e4f2"
format_stamp: "Formatted at 2026-08-10 19:26:08 on dist-test-slave-2j7r"
server_key: "fbdd5f39780b61f848555ef65ce3f59f"
server_key_iv: "aec53b23071250e316473e79440da15a"
server_key_version: "encryptionkey@0"
I20260810 19:26:08.612552 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:08.629786 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:08.633476 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:08.633716 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:08.634692 27158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:08.636026 27158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:08.636111 27158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.636196 27158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:08.636252 27158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:08.666162 27158 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:44281
I20260810 19:26:08.666214 27385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:44281 every 8 connection(s)
I20260810 19:26:08.666996 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:08.672966 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27158
I20260810 19:26:08.673164 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:08.673604 19728 external_mini_cluster.cc:1428] Setting key d1f7751352214bd2627f74dc76c9dfb5
Aug 10 19:26:08 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389968, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:08.687690 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:08.669394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:56717 (local address 127.19.68.62:40087)
0810 19:26:08.669597 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:08.669604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:08.670762 (+ 1158us) server_negotiation.cc:407] Connection header received
0810 19:26:08.672588 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:08.672594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:08.672683 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:08.672800 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:08.674320 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.675315 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.677395 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.677833 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.677931 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:08.682024 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:08.682028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:08.682032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:08.682066 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:08.684603 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:08.685470 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:08.685474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:08.685476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:08.685606 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:08.686022 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:08.686026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:08.686028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:08.686301 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:08.686435 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:08.687116 (+ 681us) server_negotiation.cc:299] Negotiation successful
0810 19:26:08.687350 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:26:08.688709 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:08.669905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40087 (local address 127.19.68.2:56717)
0810 19:26:08.670610 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:08.670648 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:08.672319 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:08.672962 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:08.672970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:08.673564 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:08.674140 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:08.674161 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.675459 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.675463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:08.677247 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:08.677263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:08.678054 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:08.678061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:08.678308 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:08.679074 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:08.679101 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:08.681765 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:08.684750 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:08.684758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:08.684773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:08.685329 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:08.685710 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:08.685714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:08.685716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:08.685895 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:08.686428 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:08.686436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:08.686757 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 19:26:08.686993 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:26:08.687534 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260810 19:26:08.690236 27388 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40087
I20260810 19:26:08.690569 27388 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:08.691409 27388 heartbeater.cc:511] Master 127.19.68.62:40087 requested a full tablet report, sending...
I20260810 19:26:08.692930 26835 ts_manager.cc:194] Registered new tserver with Master: aa7ed972452b4962ae82dcf95139e4f2 (127.19.68.2:44281)
I20260810 19:26:08.693929 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:56717
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:26:08.727778 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:09.047895 27395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:09.048141 27395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:09.048244 27395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:09.055274 27395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:09.055377 27395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:09.055497 27395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:09.064954 27395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:09.066224 27395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:09.067530 27395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:09.080025 27402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:09.079984 27403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:09.080050 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:09.080816 27395 server_base.cc:1034] running on GCE node
I20260810 19:26:09.081453 27395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:09.082357 27395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:09.083567 27395 hybrid_clock.cc:648] HybridClock initialized: now 1786389969083517 us; error 36 us; skew 500 ppm
I20260810 19:26:09.083873 27395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:09 dist-test-slave-2j7r krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389969, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:09.086884 27395 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:26:09.087947 27395 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:09.088048 27395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:09.088338 27395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:09.091403 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "0a00cb2dfcba4202a69f7025bef41d8c"
format_stamp: "Formatted at 2026-08-10 19:26:09 on dist-test-slave-2j7r"
server_key: "76c69a723ab612f9f644b0358c97c957"
server_key_iv: "f5a9f38684b7700fd0f27c832795d827"
server_key_version: "encryptionkey@0"
I20260810 19:26:09.092159 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "0a00cb2dfcba4202a69f7025bef41d8c"
format_stamp: "Formatted at 2026-08-10 19:26:09 on dist-test-slave-2j7r"
server_key: "76c69a723ab612f9f644b0358c97c957"
server_key_iv: "f5a9f38684b7700fd0f27c832795d827"
server_key_version: "encryptionkey@0"
I20260810 19:26:09.097024 27395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:26:09.100296 27422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:09.101572 27395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:09.101709 27395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "0a00cb2dfcba4202a69f7025bef41d8c"
format_stamp: "Formatted at 2026-08-10 19:26:09 on dist-test-slave-2j7r"
server_key: "76c69a723ab612f9f644b0358c97c957"
server_key_iv: "f5a9f38684b7700fd0f27c832795d827"
server_key_version: "encryptionkey@0"
I20260810 19:26:09.101902 27395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:09.119565 27395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:09.123471 27395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:09.123718 27395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:09.124656 27395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:09.126062 27395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:09.126156 27395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:09.126266 27395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:09.126335 27395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:09.156229 27395 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42435
I20260810 19:26:09.156278 27630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42435 every 8 connection(s)
I20260810 19:26:09.157034 27395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:09.159650 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27395
I20260810 19:26:09.159843 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:09.160272 19728 external_mini_cluster.cc:1428] Setting key 5cecb058109c38d3dc6e9a1fa6bde37d
Aug 10 19:26:09 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389969, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:09.177203 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.159408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:46845 (local address 127.19.68.62:40087)
0810 19:26:09.159639 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.159645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.160896 (+ 1251us) server_negotiation.cc:407] Connection header received
0810 19:26:09.163177 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.163182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.163289 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.163370 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:09.164870 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.165646 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.167111 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.167509 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.167602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.170904 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:09.170910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:09.170915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:09.170955 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:09.174273 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.175148 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.175153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.175157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.175324 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.175796 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.175800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.175802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.176073 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:09.176183 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.176765 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.176922 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:26:09.178073 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.159900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40087 (local address 127.19.68.3:46845)
0810 19:26:09.160706 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.160768 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.162879 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.163593 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.163601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.164117 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:09.164702 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.164725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.165759 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.165763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.166991 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.167004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.167674 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.167678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.167854 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.168517 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:09.168544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:09.170691 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:09.174441 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:09.174449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:09.174463 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:09.175029 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:09.175454 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:09.175459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:09.175462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:09.175690 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:09.176168 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:09.176173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:09.176473 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.176690 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.177053 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260810 19:26:09.179237 27631 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40087
I20260810 19:26:09.179519 27631 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:09.180135 27631 heartbeater.cc:511] Master 127.19.68.62:40087 requested a full tablet report, sending...
I20260810 19:26:09.181532 26835 ts_manager.cc:194] Registered new tserver with Master: 0a00cb2dfcba4202a69f7025bef41d8c (127.19.68.3:42435)
I20260810 19:26:09.182297 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:46845
I20260810 19:26:09.187467 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:26:09.203933 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.191940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.19.68.62:40087)
0810 19:26:09.192089 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.192093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.192561 (+ 468us) server_negotiation.cc:407] Connection header received
0810 19:26:09.192702 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.192706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.192774 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.192845 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:09.194273 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.195036 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.196295 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.196847 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.196946 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.199042 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:09.199046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:09.199050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:09.199082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:09.201539 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.202160 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.202164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.202167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.202343 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.202663 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.202666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.202668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.202986 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:09.203175 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.203473 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.203639 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:26:09.204077 27142 heartbeater.cc:503] Master 127.19.68.62:40087 was elected leader, sending a full tablet report...
I20260810 19:26:09.208348 26834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:09.215250 26834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:09.241879 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.229111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.19.68.3:42435)
0810 19:26:09.234273 (+ 5162us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.234281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.234310 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:26:09.234403 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.234410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.234615 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.234798 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:09.235686 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.237244 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.239428 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.240749 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.240867 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.241132 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.241404 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.241560 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4999,"thread_start_us":4837,"threads_started":1}
I20260810 19:26:09.241941 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.228477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42435 (local address 127.0.0.1:54050)
0810 19:26:09.229510 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.229527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.229632 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.234810 (+ 5178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.234814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.234848 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:09.235446 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.235460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.237379 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.237384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.239269 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.239286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.240939 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.240945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.241153 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.241206 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.241328 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.241517 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":183,"thread_start_us":804,"threads_started":1}
I20260810 19:26:09.242486 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.227788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44281 (local address 127.0.0.1:35746)
0810 19:26:09.228697 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.228739 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.228916 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.234517 (+ 5601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.234525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.234558 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:09.235157 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.235175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.237131 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.237137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.239795 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.239819 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.241669 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.241675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.241848 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.241880 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.241956 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.242091 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":212,"thread_start_us":469,"threads_started":1}
I20260810 19:26:09.243023 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.229815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.19.68.1:44285)
0810 19:26:09.230595 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.230605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.233701 (+ 3096us) server_negotiation.cc:407] Connection header received
0810 19:26:09.233999 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.234008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.234266 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.234406 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:09.235625 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.236976 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.240204 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.241650 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.241763 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.241989 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.242525 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.242695 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":381,"threads_started":1}
I20260810 19:26:09.243460 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.228048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.19.68.2:44281)
0810 19:26:09.233876 (+ 5828us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.233885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.233916 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:26:09.234027 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.234034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.234272 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.234492 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:09.235359 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.236974 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.239995 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.241416 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.242733 (+ 1317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.242925 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.242998 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.243151 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5684,"thread_start_us":5541,"threads_started":1}
I20260810 19:26:09.243810 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.229186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44285 (local address 127.0.0.1:42694)
0810 19:26:09.233570 (+ 4384us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.233589 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.233811 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.234716 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.234720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.234742 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:09.235291 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.235314 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.237158 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.237162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.240039 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.240051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.242127 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.242132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.242297 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.242339 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.243337 (+ 998us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.243466 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4243,"mutex_wait_us":115,"thread_start_us":4085,"threads_started":1}
I20260810 19:26:09.246201 27513 tablet_service.cc:1467] Processing CreateTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 (DEFAULT_TABLE table=test-table [id=2a9a166192164959b806aa8acc197c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:09.246484 27270 tablet_service.cc:1467] Processing CreateTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 (DEFAULT_TABLE table=test-table [id=2a9a166192164959b806aa8acc197c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:09.247635 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf553fa4cb014a07829e15de4e9e7a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:09.247205 27038 tablet_service.cc:1467] Processing CreateTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 (DEFAULT_TABLE table=test-table [id=2a9a166192164959b806aa8acc197c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:09.247715 27513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf553fa4cb014a07829e15de4e9e7a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:09.248502 27038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf553fa4cb014a07829e15de4e9e7a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:09.258466 27663 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Bootstrap starting.
I20260810 19:26:09.261695 27664 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Bootstrap starting.
I20260810 19:26:09.262648 27663 tablet_bootstrap.cc:654] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:09.263931 27663 log.cc:824] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:09.265233 27664 tablet_bootstrap.cc:654] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:09.265383 27665 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Bootstrap starting.
I20260810 19:26:09.266912 27663 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: No bootstrap required, opened a new log
I20260810 19:26:09.266574 27664 log.cc:824] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:09.267177 27663 ts_tablet_manager.cc:1397] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:26:09.268585 27665 tablet_bootstrap.cc:654] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:09.269562 27665 log.cc:824] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:09.272361 27664 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: No bootstrap required, opened a new log
I20260810 19:26:09.272534 27665 tablet_bootstrap.cc:492] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: No bootstrap required, opened a new log
I20260810 19:26:09.272609 27664 ts_tablet_manager.cc:1397] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260810 19:26:09.272787 27665 ts_tablet_manager.cc:1397] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 19:26:09.277318 27663 raft_consensus.cc:348] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.277758 27663 raft_consensus.cc:374] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:09.277904 27663 raft_consensus.cc:729] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7ed972452b4962ae82dcf95139e4f2, State: Initialized, Role: FOLLOWER
I20260810 19:26:09.278532 27663 consensus_queue.cc:260] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.279676 27388 heartbeater.cc:503] Master 127.19.68.62:40087 was elected leader, sending a full tablet report...
I20260810 19:26:09.280252 27663 ts_tablet_manager.cc:1428] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 19:26:09.281432 27664 raft_consensus.cc:348] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.281759 27664 raft_consensus.cc:374] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:09.281868 27664 raft_consensus.cc:729] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a00cb2dfcba4202a69f7025bef41d8c, State: Initialized, Role: FOLLOWER
I20260810 19:26:09.282405 27664 consensus_queue.cc:260] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.282445 27665 raft_consensus.cc:348] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.282904 27665 raft_consensus.cc:374] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:09.283036 27665 raft_consensus.cc:729] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c24eecff54e529aa4f3d25efe1988, State: Initialized, Role: FOLLOWER
I20260810 19:26:09.283506 27631 heartbeater.cc:503] Master 127.19.68.62:40087 was elected leader, sending a full tablet report...
I20260810 19:26:09.283715 27665 consensus_queue.cc:260] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.284103 27664 ts_tablet_manager.cc:1428] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260810 19:26:09.285295 27665 ts_tablet_manager.cc:1428] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Time spent starting tablet: real 0.012s user 0.010s sys 0.005s
I20260810 19:26:09.348457 27675 raft_consensus.cc:482] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:09.348767 27675 raft_consensus.cc:504] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.350476 27675 leader_election.cc:288] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa7ed972452b4962ae82dcf95139e4f2 (127.19.68.2:44281), e92c24eecff54e529aa4f3d25efe1988 (127.19.68.1:44285)
I20260810 19:26:09.355948 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.350921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:60515 (local address 127.19.68.2:44281)
0810 19:26:09.351099 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.351103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.351127 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:26:09.351376 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.351382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.351464 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.351564 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:09.352005 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.352721 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.354335 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.355110 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.355229 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.355323 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.355586 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.355709 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:26:09.355952 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.350834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44281 (local address 127.19.68.3:60515)
0810 19:26:09.351058 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.351077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.351204 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.351610 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.351615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.351645 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:09.351849 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.351866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.352864 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.352868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.354173 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.354186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.355269 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.355275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.355441 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.355467 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.355548 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.355671 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 19:26:09.356985 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.351262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44285 (local address 127.19.68.3:49795)
0810 19:26:09.352101 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:09.352117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:09.352266 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:09.352586 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:09.352591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:09.352616 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:09.352947 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.352961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.353984 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.353987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.355276 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:09.355289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.356415 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.356421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:09.356576 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.356604 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:26:09.356675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 19:26:09.356756 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":536,"threads_started":1}
I20260810 19:26:09.357131 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.351412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:49795 (local address 127.19.68.1:44285)
0810 19:26:09.351641 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.351648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.352203 (+ 555us) server_negotiation.cc:407] Connection header received
0810 19:26:09.352412 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.352418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.352480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.352550 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:09.353123 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.353835 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.355425 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.356219 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.356342 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.356429 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.356737 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.356873 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:26:09.357066 27305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf553fa4cb014a07829e15de4e9e7a57" candidate_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7ed972452b4962ae82dcf95139e4f2" is_pre_election: true
I20260810 19:26:09.357710 27305 raft_consensus.cc:2454] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a00cb2dfcba4202a69f7025bef41d8c in term 0.
I20260810 19:26:09.357959 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf553fa4cb014a07829e15de4e9e7a57" candidate_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c24eecff54e529aa4f3d25efe1988" is_pre_election: true
I20260810 19:26:09.358395 27424 leader_election.cc:302] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a00cb2dfcba4202a69f7025bef41d8c, aa7ed972452b4962ae82dcf95139e4f2; no voters:
I20260810 19:26:09.358457 27069 raft_consensus.cc:2454] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a00cb2dfcba4202a69f7025bef41d8c in term 0.
I20260810 19:26:09.358770 27675 raft_consensus.cc:2781] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:09.358932 27675 raft_consensus.cc:482] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:09.359042 27675 raft_consensus.cc:3037] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:09.361449 27675 raft_consensus.cc:504] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.361987 27675 leader_election.cc:288] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [CANDIDATE]: Term 1 election: Requested vote from peers aa7ed972452b4962ae82dcf95139e4f2 (127.19.68.2:44281), e92c24eecff54e529aa4f3d25efe1988 (127.19.68.1:44285)
I20260810 19:26:09.362423 27305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf553fa4cb014a07829e15de4e9e7a57" candidate_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7ed972452b4962ae82dcf95139e4f2"
I20260810 19:26:09.362478 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf553fa4cb014a07829e15de4e9e7a57" candidate_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c24eecff54e529aa4f3d25efe1988"
I20260810 19:26:09.362608 27305 raft_consensus.cc:3037] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:09.362658 27069 raft_consensus.cc:3037] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:09.364796 27069 raft_consensus.cc:2454] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a00cb2dfcba4202a69f7025bef41d8c in term 1.
I20260810 19:26:09.364821 27305 raft_consensus.cc:2454] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a00cb2dfcba4202a69f7025bef41d8c in term 1.
I20260810 19:26:09.365170 27424 leader_election.cc:302] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a00cb2dfcba4202a69f7025bef41d8c, e92c24eecff54e529aa4f3d25efe1988; no voters:
I20260810 19:26:09.365553 27675 raft_consensus.cc:2781] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:09.366020 27675 raft_consensus.cc:686] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 1 LEADER]: Becoming Leader. State: Replica: 0a00cb2dfcba4202a69f7025bef41d8c, State: Running, Role: LEADER
I20260810 19:26:09.366788 27675 consensus_queue.cc:237] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } }
I20260810 19:26:09.371533 26832 catalog_manager.cc:5495] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a00cb2dfcba4202a69f7025bef41d8c (127.19.68.3). New cstate: current_term: 1 leader_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a00cb2dfcba4202a69f7025bef41d8c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42435 } health_report { overall_health: HEALTHY } } }
I20260810 19:26:09.403167 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.398533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.19.68.3:42435)
0810 19:26:09.398717 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.398722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.398740 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:26:09.398833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.398837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.398923 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.399044 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:09.399334 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.400073 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.401374 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.401791 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.401886 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.402276 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:09.402750 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:09.402873 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.402912 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.402974 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:26:09.408373 27632 tablet.cc:2369] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:09.411432 27069 raft_consensus.cc:1261] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 1 FOLLOWER]: Refusing update from remote peer 0a00cb2dfcba4202a69f7025bef41d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:09.411440 27305 raft_consensus.cc:1261] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 1 FOLLOWER]: Refusing update from remote peer 0a00cb2dfcba4202a69f7025bef41d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:09.412339 27678 consensus_queue.cc:1035] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ed972452b4962ae82dcf95139e4f2" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:09.412678 27675 consensus_queue.cc:1035] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c24eecff54e529aa4f3d25efe1988" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:26:09.418807 27389 tablet.cc:2369] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:09.421846 27682 mvcc.cc:205] Tried to move back new op lower bound from 7317053314699653120 to 7317053314529992704. Current Snapshot: MvccSnapshot[applied={T|T < 7317053314699653120}]
W20260810 19:26:09.422070 27143 tablet.cc:2369] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:09.425967 27683 mvcc.cc:205] Tried to move back new op lower bound from 7317053314699653120 to 7317053314529992704. Current Snapshot: MvccSnapshot[applied={T|T < 7317053314699653120}]
I20260810 19:26:09.437829 26832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:09.438117 26832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:09.441622 26832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a77f19f05fb14174a379638dd19dfbd4: Sending DeleteTablet for 3 replicas of tablet bf553fa4cb014a07829e15de4e9e7a57
I20260810 19:26:09.442875 27513 tablet_service.cc:1514] Processing DeleteTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:54050
I20260810 19:26:09.443004 27270 tablet_service.cc:1514] Processing DeleteTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260810 19:26:09.443815 27693 tablet_replica.cc:330] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: stopping tablet replica
I20260810 19:26:09.444049 27038 tablet_service.cc:1514] Processing DeleteTablet for tablet bf553fa4cb014a07829e15de4e9e7a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:42694
I20260810 19:26:09.444322 27693 raft_consensus.cc:2229] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:09.444446 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26935
I20260810 19:26:09.445178 27693 raft_consensus.cc:2258] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:09.447443 27693 ts_tablet_manager.cc:1905] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:09.455371 27693 ts_tablet_manager.cc:1918] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:09.455626 27693 log.cc:1196] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/bf553fa4cb014a07829e15de4e9e7a57
I20260810 19:26:09.456293 27693 ts_tablet_manager.cc:1939] T bf553fa4cb014a07829e15de4e9e7a57 P 0a00cb2dfcba4202a69f7025bef41d8c: Deleting consensus metadata
I20260810 19:26:09.458158 26815 catalog_manager.cc:4841] TS 0a00cb2dfcba4202a69f7025bef41d8c (127.19.68.3:42435): tablet bf553fa4cb014a07829e15de4e9e7a57 (table test-table [id=2a9a166192164959b806aa8acc197c5d]) successfully deleted
I20260810 19:26:09.458772 27696 tablet_replica.cc:330] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: stopping tablet replica
I20260810 19:26:09.459143 27696 raft_consensus.cc:2229] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:09.459563 27696 raft_consensus.cc:2258] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:09.460754 27694 tablet_replica.cc:330] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: stopping tablet replica
I20260810 19:26:09.461211 27694 raft_consensus.cc:2229] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:09.461710 27694 raft_consensus.cc:2258] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:09.462184 27696 ts_tablet_manager.cc:1905] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:09.463896 27694 ts_tablet_manager.cc:1905] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:26:09 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:09.470103 27696 ts_tablet_manager.cc:1918] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:09.470419 27696 log.cc:1196] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/bf553fa4cb014a07829e15de4e9e7a57
I20260810 19:26:09.471138 27696 ts_tablet_manager.cc:1939] T bf553fa4cb014a07829e15de4e9e7a57 P e92c24eecff54e529aa4f3d25efe1988: Deleting consensus metadata
I20260810 19:26:09.471464 27694 ts_tablet_manager.cc:1918] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:09.471681 27694 log.cc:1196] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/bf553fa4cb014a07829e15de4e9e7a57
I20260810 19:26:09.472239 27694 ts_tablet_manager.cc:1939] T bf553fa4cb014a07829e15de4e9e7a57 P aa7ed972452b4962ae82dcf95139e4f2: Deleting consensus metadata
I20260810 19:26:09.473496 26810 catalog_manager.cc:4841] TS e92c24eecff54e529aa4f3d25efe1988 (127.19.68.1:44285): tablet bf553fa4cb014a07829e15de4e9e7a57 (table test-table [id=2a9a166192164959b806aa8acc197c5d]) successfully deleted
I20260810 19:26:09.474040 26810 catalog_manager.cc:4841] TS aa7ed972452b4962ae82dcf95139e4f2 (127.19.68.2:44281): tablet bf553fa4cb014a07829e15de4e9e7a57 (table test-table [id=2a9a166192164959b806aa8acc197c5d]) successfully deleted
I20260810 19:26:09.476212 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.446730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.19.68.1:44285)
0810 19:26:09.446908 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.446915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.455854 (+ 8939us) server_negotiation.cc:407] Connection header received
0810 19:26:09.456329 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.456335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.456455 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.456535 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:09.460218 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.461322 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.463102 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.463761 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.463902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.469153 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:09.469169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:09.469187 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:09.469232 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:09.473264 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.474146 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.474156 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.474159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.474312 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.474687 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.474693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.474696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.475061 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:09.475373 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.475666 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.475882 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:26:09.476927 27136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:09.577812 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26935
I20260810 19:26:09.602784 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27158
Aug 10 19:26:09 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:26:09.619284 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.603625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.19.68.2:44281)
0810 19:26:09.603837 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.603844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.603865 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:26:09.603929 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.603934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.604034 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.604119 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:09.605611 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.606545 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.608375 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.608749 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.608842 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.613694 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:09.613710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:09.613725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:09.613760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:09.616544 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.617232 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.617240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.617242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.617378 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.617738 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.617741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.617743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.618143 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:09.618429 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.618750 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.618967 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:26:09.619915 27380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:09.709195 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27158
I20260810 19:26:09.733542 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27395
Aug 10 19:26:09 dist-test-slave-2j7r krb5kdc[26763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:26:09.751338 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:09.734437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.19.68.3:42435)
0810 19:26:09.734678 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:09.734685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:09.734727 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:26:09.734805 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:09.734812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:09.734901 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:09.735028 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:09.736303 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.737728 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.739079 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:09.739801 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:09.739958 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:09.744216 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:09.744242 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:09.744263 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:09.744312 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:09.748193 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.748839 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.748854 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.748858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.749075 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:09.749476 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:09.749482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:09.749485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:09.750030 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:09.750424 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:09.750590 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 19:26:09.750844 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 19:26:09.752128 27621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:09.876332 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27395
I20260810 19:26:09.902967 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26783
I20260810 19:26:09.904107 26885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:10.012333 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26783
2026-08-10T19:26:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4356 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T19:26:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:10Z Disabled control of system clock
I20260810 19:26:10.090997 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:43431
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 19:26:10.408902 27709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:10.409191 27709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:10.409284 27709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:10.416257 27709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:10.416363 27709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:10.416407 27709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:10.416461 27709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:10.416504 27709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:10.425273 27709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:43431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:10.426486 27709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:10.428175 27709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:10.439450 27719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:10.439396 27721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:10.439404 27723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:10.440506 27709 server_base.cc:1034] running on GCE node
I20260810 19:26:10.441290 27709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:10.442678 27709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:10.443884 27709 hybrid_clock.cc:648] HybridClock initialized: now 1786389970443836 us; error 45 us; skew 500 ppm
I20260810 19:26:10.444208 27709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:10.446317 27709 webserver.cc:460] Webserver started at http://127.19.68.62:33125/ using document root <none> and password file <none>
I20260810 19:26:10.446964 27709 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:10.447065 27709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:10.447365 27709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:10.450166 27709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "371aa7748f0541d899c4ed9011372c4e"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "55afa4ef94038722498dcecfa8b956db"
server_key_iv: "dce1a09b357a06d462996afdfc190964"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.450917 27709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "371aa7748f0541d899c4ed9011372c4e"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "55afa4ef94038722498dcecfa8b956db"
server_key_iv: "dce1a09b357a06d462996afdfc190964"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.455682 27709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 19:26:10.458869 27736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:10.460140 27709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 19:26:10.460275 27709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "371aa7748f0541d899c4ed9011372c4e"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "55afa4ef94038722498dcecfa8b956db"
server_key_iv: "dce1a09b357a06d462996afdfc190964"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.460459 27709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:10.465830 27709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:10.466650 27709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:10.466888 27709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:10.485836 27709 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:43431
I20260810 19:26:10.485881 27818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:43431 every 8 connection(s)
I20260810 19:26:10.487277 27709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:10.489877 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27709
I20260810 19:26:10.490043 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:10.490685 19728 external_mini_cluster.cc:1428] Setting key 7f858ec5be29ad0863a7e4e582937cf1
I20260810 19:26:10.491115 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:10.499884 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:10.492633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.19.68.62:43431)
0810 19:26:10.493757 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:10.493773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:10.493826 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:26:10.495853 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:10.495879 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:10.496403 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:10.496851 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:10.497928 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:10.497936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:10.497952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:10.497969 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:10.498051 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:10.498075 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:10.498307 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:10.498450 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 19:26:10.498710 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":371,"threads_started":1}
I20260810 19:26:10.504611 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: Bootstrap starting.
I20260810 19:26:10.507432 27821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:10.508383 27821 log.cc:824] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:10.510936 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: No bootstrap required, opened a new log
I20260810 19:26:10.517817 27821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } }
I20260810 19:26:10.518134 27821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:10.518240 27821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371aa7748f0541d899c4ed9011372c4e, State: Initialized, Role: FOLLOWER
I20260810 19:26:10.518823 27821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } }
I20260810 19:26:10.519028 27821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:10.519155 27821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:10.519292 27821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:10.521360 27821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } }
I20260810 19:26:10.521804 27821 leader_election.cc:302] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 371aa7748f0541d899c4ed9011372c4e; no voters:
I20260810 19:26:10.522461 27821 leader_election.cc:288] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:10.522588 27828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:10.524155 27828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [term 1 LEADER]: Becoming Leader. State: Replica: 371aa7748f0541d899c4ed9011372c4e, State: Running, Role: LEADER
I20260810 19:26:10.524596 27821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:10.524827 27828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } }
I20260810 19:26:10.527623 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371aa7748f0541d899c4ed9011372c4e. Latest consensus state: current_term: 1 leader_uuid: "371aa7748f0541d899c4ed9011372c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } } }
I20260810 19:26:10.528086 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:10.528062 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "371aa7748f0541d899c4ed9011372c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371aa7748f0541d899c4ed9011372c4e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43431 } } }
I20260810 19:26:10.528712 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:10.529193 27845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:10.534301 27845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:10.542513 27845 catalog_manager.cc:1347] Generated new cluster ID: 5f239aa011e34bbabd561aa2437c8e86
I20260810 19:26:10.542611 27845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:10.560074 27845 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:10.560459 27845 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:10.566517 27845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: Generated new TSK 0
I20260810 19:26:10.567221 27845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:26:10.573700 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--builtin_ntp_servers=127.19.68.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 19:26:10.896905 27860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:10.897192 27860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:10.897343 27860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:10.904335 27860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:10.904420 27860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:10.904524 27860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:10.913220 27860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:10.914381 27860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:10.916138 27860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:10.928877 27866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:10.928817 27867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:10.928903 27869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:10.929723 27860 server_base.cc:1034] running on GCE node
I20260810 19:26:10.930315 27860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:10.931281 27860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:10.932557 27860 hybrid_clock.cc:648] HybridClock initialized: now 1786389970932518 us; error 39 us; skew 500 ppm
I20260810 19:26:10.932891 27860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:10.935250 27860 webserver.cc:460] Webserver started at http://127.19.68.1:45775/ using document root <none> and password file <none>
I20260810 19:26:10.935999 27860 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:10.936110 27860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:10.936446 27860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:10.939333 27860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "4c26baa558164def9c83dc987661f5fb"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "c013b2b0a22d51fc859b7ab352481677"
server_key_iv: "1d6d3fcd0b707746ddaf3c9e5561fc30"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.940119 27860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "4c26baa558164def9c83dc987661f5fb"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "c013b2b0a22d51fc859b7ab352481677"
server_key_iv: "1d6d3fcd0b707746ddaf3c9e5561fc30"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.945333 27860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 19:26:10.948957 27878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:10.950294 27860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:26:10.950462 27860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "4c26baa558164def9c83dc987661f5fb"
format_stamp: "Formatted at 2026-08-10 19:26:10 on dist-test-slave-2j7r"
server_key: "c013b2b0a22d51fc859b7ab352481677"
server_key_iv: "1d6d3fcd0b707746ddaf3c9e5561fc30"
server_key_version: "encryptionkey@0"
I20260810 19:26:10.950660 27860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:10.956019 27860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:10.956746 27860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:10.956983 27860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:10.957870 27860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:10.959175 27860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:10.959267 27860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:10.959378 27860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:10.959448 27860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:10.987879 27860 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:39893
I20260810 19:26:10.987931 28079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:39893 every 8 connection(s)
I20260810 19:26:10.989110 27860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:10.996555 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:10.991535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:57795 (local address 127.19.68.62:43431)
0810 19:26:10.991734 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:10.991740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:10.993071 (+ 1331us) server_negotiation.cc:407] Connection header received
0810 19:26:10.994706 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:10.994710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:10.994769 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:10.994866 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:10.995946 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:10.995950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:10.995953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:10.995968 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:10.996001 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:10.996013 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:10.996101 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:10.996294 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:26:10.996342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:26:10.997128 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27860
I20260810 19:26:10.997298 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:10.997943 19728 external_mini_cluster.cc:1428] Setting key ea39989a88077bd6afb1509978623c5d
I20260810 19:26:10.998008 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:10.992015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:43431 (local address 127.19.68.1:57795)
0810 19:26:10.992923 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:10.992966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:10.994472 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:10.995024 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:10.995031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:10.995112 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:10.995670 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:10.995672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:10.995708 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:10.995801 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:10.996123 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:10.996132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:10.996139 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:26:10.996373 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 19:26:10.996553 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":293,"threads_started":1}
I20260810 19:26:10.999570 28080 heartbeater.cc:347] Connected to a master server at 127.19.68.62:43431
I20260810 19:26:10.999850 28080 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:11.000499 28080 heartbeater.cc:511] Master 127.19.68.62:43431 requested a full tablet report, sending...
I20260810 19:26:11.001197 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--builtin_ntp_servers=127.19.68.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 19:26:11.002548 27762 ts_manager.cc:194] Registered new tserver with Master: 4c26baa558164def9c83dc987661f5fb (127.19.68.1:39893)
W20260810 19:26:11.320672 28083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:11.320936 28083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:11.321023 28083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:11.328140 28083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:11.328233 28083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:11.328348 28083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:11.337114 28083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:11.338284 28083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:11.339946 28083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:11.352187 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:11.352137 28095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:11.352214 28091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:11.353186 28083 server_base.cc:1034] running on GCE node
I20260810 19:26:11.353818 28083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:11.354691 28083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:11.355950 28083 hybrid_clock.cc:648] HybridClock initialized: now 1786389971355882 us; error 55 us; skew 500 ppm
I20260810 19:26:11.356261 28083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:11.358515 28083 webserver.cc:460] Webserver started at http://127.19.68.2:39705/ using document root <none> and password file <none>
I20260810 19:26:11.359143 28083 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:11.359233 28083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:11.359544 28083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:11.362339 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "ddd39b0771f04736bf4f930e85572137"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "1b044fd18925a966321694d0bc2e0a83"
server_key_iv: "b8ec0ab8520a5b9eb212ee12bf18dac0"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.363103 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "ddd39b0771f04736bf4f930e85572137"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "1b044fd18925a966321694d0bc2e0a83"
server_key_iv: "b8ec0ab8520a5b9eb212ee12bf18dac0"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.368014 28083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 19:26:11.371243 28104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.372438 28083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:26:11.372592 28083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "ddd39b0771f04736bf4f930e85572137"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "1b044fd18925a966321694d0bc2e0a83"
server_key_iv: "b8ec0ab8520a5b9eb212ee12bf18dac0"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.372781 28083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:11.378123 28083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:11.378849 28083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:11.379088 28083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:11.379948 28083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:11.381345 28083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:11.381439 28083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.381542 28083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:11.381611 28083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.412642 28083 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:45629
I20260810 19:26:11.412690 28301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:45629 every 8 connection(s)
I20260810 19:26:11.413985 28083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:11.421882 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.416509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:38211 (local address 127.19.68.62:43431)
0810 19:26:11.416717 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.416724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.418170 (+ 1446us) server_negotiation.cc:407] Connection header received
0810 19:26:11.419928 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.419932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.419991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.420081 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.421293 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.421297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.421300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.421317 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.421372 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.421384 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.421478 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.421599 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.421653 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:26:11.422762 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.417050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:43431 (local address 127.19.68.2:38211)
0810 19:26:11.418004 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:11.418044 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:11.419649 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:11.420263 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:11.420270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:11.420371 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:11.420965 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:11.420967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:11.421003 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:11.421118 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:11.421452 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:11.421458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:11.421462 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:26:11.421660 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 19:26:11.421850 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":264,"threads_started":1}
I20260810 19:26:11.424036 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 28083
I20260810 19:26:11.424088 28302 heartbeater.cc:347] Connected to a master server at 127.19.68.62:43431
I20260810 19:26:11.424189 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:11.424381 28302 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:11.424726 19728 external_mini_cluster.cc:1428] Setting key 312e65fba30f834c183cbefa960420a9
I20260810 19:26:11.424991 28302 heartbeater.cc:511] Master 127.19.68.62:43431 requested a full tablet report, sending...
I20260810 19:26:11.426381 27763 ts_manager.cc:194] Registered new tserver with Master: ddd39b0771f04736bf4f930e85572137 (127.19.68.2:45629)
I20260810 19:26:11.427718 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--builtin_ntp_servers=127.19.68.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 19:26:11.751121 28306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:11.751365 28306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:11.751458 28306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:11.758430 28306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:11.758517 28306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:11.758630 28306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:11.767462 28306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:11.768674 28306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:11.770485 28306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:11.783061 28317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:11.783030 28324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:11.783082 28320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:11.784518 28306 server_base.cc:1034] running on GCE node
I20260810 19:26:11.785056 28306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:11.785988 28306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:11.787173 28306 hybrid_clock.cc:648] HybridClock initialized: now 1786389971787123 us; error 46 us; skew 500 ppm
I20260810 19:26:11.787457 28306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:11.789588 28306 webserver.cc:460] Webserver started at http://127.19.68.3:34263/ using document root <none> and password file <none>
I20260810 19:26:11.790241 28306 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:11.790328 28306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:11.790606 28306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:11.793430 28306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "5e31feb5605b4b9585a2b60bfc8aa48c"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "95c4410164ab1aa281f99af66df47ae8"
server_key_iv: "46066d87830bf74a4f7e97a27e5bcee3"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.794164 28306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "5e31feb5605b4b9585a2b60bfc8aa48c"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "95c4410164ab1aa281f99af66df47ae8"
server_key_iv: "46066d87830bf74a4f7e97a27e5bcee3"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.799172 28306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:11.802788 28333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.804078 28306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:26:11.804230 28306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "5e31feb5605b4b9585a2b60bfc8aa48c"
format_stamp: "Formatted at 2026-08-10 19:26:11 on dist-test-slave-2j7r"
server_key: "95c4410164ab1aa281f99af66df47ae8"
server_key_iv: "46066d87830bf74a4f7e97a27e5bcee3"
server_key_version: "encryptionkey@0"
I20260810 19:26:11.804440 28306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:11.810007 28306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:11.810779 28306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:11.811004 28306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:11.811877 28306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:11.813259 28306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:11.813345 28306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.813432 28306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:11.813488 28306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:11.843713 28306 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:36225
I20260810 19:26:11.843763 28519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:36225 every 8 connection(s)
I20260810 19:26:11.844965 28306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:11.851127 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 28306
I20260810 19:26:11.851298 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:11.851840 19728 external_mini_cluster.cc:1428] Setting key bfee6b2b4e813088abd3b0dc47de50c2
I20260810 19:26:11.853140 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.847442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:55097 (local address 127.19.68.62:43431)
0810 19:26:11.847638 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.847642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.848921 (+ 1279us) server_negotiation.cc:407] Connection header received
0810 19:26:11.850655 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.850660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.850718 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.850821 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.852235 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.852239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.852243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.852259 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.852294 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.852306 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.852406 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.852797 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.852848 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:11.854182 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.847921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:43431 (local address 127.19.68.3:55097)
0810 19:26:11.848786 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:11.848821 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:11.850382 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:11.851090 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:11.851102 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:11.851220 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:11.851890 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:11.851894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:11.851947 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:11.852059 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:11.852597 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:11.852606 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:11.852613 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:26:11.852869 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 19:26:11.853139 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":249,"threads_started":1}
I20260810 19:26:11.855497 28520 heartbeater.cc:347] Connected to a master server at 127.19.68.62:43431
I20260810 19:26:11.855762 28520 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:11.856314 28520 heartbeater.cc:511] Master 127.19.68.62:43431 requested a full tablet report, sending...
I20260810 19:26:11.857615 27763 ts_manager.cc:194] Registered new tserver with Master: 5e31feb5605b4b9585a2b60bfc8aa48c (127.19.68.3:36225)
I20260810 19:26:11.866918 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:26:11.873595 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.871319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.19.68.62:43431)
0810 19:26:11.871505 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.871511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.871991 (+ 480us) server_negotiation.cc:407] Connection header received
0810 19:26:11.872130 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.872134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.872185 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.872251 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.872997 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.873001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.873004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.873018 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.873047 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.873057 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.873197 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.873262 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.873332 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:26:11.877259 27763 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:11.883682 27763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:11.900897 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.895557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.19.68.1:39893)
0810 19:26:11.896221 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.896231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.896938 (+ 707us) server_negotiation.cc:407] Connection header received
0810 19:26:11.897311 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.897322 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.897568 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.897693 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.899947 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.899956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.899969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.899998 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.900062 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.900084 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.900237 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.900471 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.900568 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":262,"threads_started":1}
I20260810 19:26:11.901335 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.895161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52662)
0810 19:26:11.896773 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:11.896819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:11.897060 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:11.898686 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:11.898699 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:11.898795 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:11.899660 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:11.899663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:11.899698 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:11.899780 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:11.900284 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:11.900288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:11.900293 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:26:11.900463 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 19:26:11.900607 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":56,"thread_start_us":1877,"threads_started":1}
I20260810 19:26:11.901425 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.895253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:45629 (local address 127.0.0.1:50410)
0810 19:26:11.897340 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:11.897356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:11.897450 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:11.898787 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:11.898792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:11.898820 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:11.899728 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:11.899731 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:11.899742 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:11.899779 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:11.900215 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:11.900227 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:11.900235 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:26:11.900437 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 19:26:11.900557 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1962,"thread_start_us":1134,"threads_started":1}
I20260810 19:26:11.902139 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.895644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.19.68.2:45629)
0810 19:26:11.897921 (+ 2277us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.897929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.897957 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:26:11.898033 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.898041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.898232 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.898370 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.899982 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.899989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.899998 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.900020 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.900062 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.900077 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.901009 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.901686 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.901790 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2115,"thread_start_us":1980,"threads_started":1}
I20260810 19:26:11.903534 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.896671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:36225 (local address 127.0.0.1:50782)
0810 19:26:11.898186 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:11.898205 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:11.898304 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:11.900227 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:11.900231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:11.900253 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:11.901007 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:11.901011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:11.901027 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:11.901066 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:11.902919 (+ 1853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:11.902924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:11.902929 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:26:11.903072 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 19:26:11.903122 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":1279,"threads_started":1}
I20260810 19:26:11.904237 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:11.897399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.19.68.3:36225)
0810 19:26:11.899756 (+ 2357us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:11.899761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:11.899787 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:26:11.899842 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:11.899848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:11.900041 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:11.900968 (+ 927us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:11.902006 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:11.902013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:11.902025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:11.902042 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:11.902089 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:11.902109 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:11.902859 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:11.903760 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 19:26:11.903854 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":1962,"threads_started":1}
I20260810 19:26:11.905004 27971 tablet_service.cc:1467] Processing CreateTablet for tablet a181084e981f499ea1c05f12da77cc31 (DEFAULT_TABLE table=test-table [id=87d811edf3f54a34b15c6011ec7ebde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:11.905638 28185 tablet_service.cc:1467] Processing CreateTablet for tablet a181084e981f499ea1c05f12da77cc31 (DEFAULT_TABLE table=test-table [id=87d811edf3f54a34b15c6011ec7ebde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:11.906600 27971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a181084e981f499ea1c05f12da77cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:11.906942 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a181084e981f499ea1c05f12da77cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:11.907737 28409 tablet_service.cc:1467] Processing CreateTablet for tablet a181084e981f499ea1c05f12da77cc31 (DEFAULT_TABLE table=test-table [id=87d811edf3f54a34b15c6011ec7ebde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:11.909122 28409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a181084e981f499ea1c05f12da77cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:11.918861 28547 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Bootstrap starting.
I20260810 19:26:11.922057 28547 tablet_bootstrap.cc:654] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:11.922926 28549 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Bootstrap starting.
I20260810 19:26:11.923408 28547 log.cc:824] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:11.925143 28548 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Bootstrap starting.
I20260810 19:26:11.926739 28549 tablet_bootstrap.cc:654] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:11.927227 28547 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: No bootstrap required, opened a new log
I20260810 19:26:11.927469 28547 ts_tablet_manager.cc:1397] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 19:26:11.927920 28549 log.cc:824] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:11.928670 28548 tablet_bootstrap.cc:654] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:11.929921 28548 log.cc:824] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:11.932524 28548 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: No bootstrap required, opened a new log
I20260810 19:26:11.932756 28548 ts_tablet_manager.cc:1397] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 19:26:11.933012 28549 tablet_bootstrap.cc:492] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: No bootstrap required, opened a new log
I20260810 19:26:11.933303 28549 ts_tablet_manager.cc:1397] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 19:26:11.936635 28547 raft_consensus.cc:348] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.937068 28547 raft_consensus.cc:374] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:11.937201 28547 raft_consensus.cc:729] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd39b0771f04736bf4f930e85572137, State: Initialized, Role: FOLLOWER
I20260810 19:26:11.937831 28547 consensus_queue.cc:260] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.938939 28302 heartbeater.cc:503] Master 127.19.68.62:43431 was elected leader, sending a full tablet report...
I20260810 19:26:11.939582 28547 ts_tablet_manager.cc:1428] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260810 19:26:11.940726 28548 raft_consensus.cc:348] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.941047 28548 raft_consensus.cc:374] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:11.941169 28548 raft_consensus.cc:729] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c26baa558164def9c83dc987661f5fb, State: Initialized, Role: FOLLOWER
I20260810 19:26:11.941699 28548 consensus_queue.cc:260] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.942688 28080 heartbeater.cc:503] Master 127.19.68.62:43431 was elected leader, sending a full tablet report...
I20260810 19:26:11.942795 28549 raft_consensus.cc:348] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.943099 28549 raft_consensus.cc:374] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:11.943162 28549 raft_consensus.cc:729] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e31feb5605b4b9585a2b60bfc8aa48c, State: Initialized, Role: FOLLOWER
I20260810 19:26:11.943343 28548 ts_tablet_manager.cc:1428] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260810 19:26:11.943660 28549 consensus_queue.cc:260] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:11.945640 28520 heartbeater.cc:503] Master 127.19.68.62:43431 was elected leader, sending a full tablet report...
I20260810 19:26:11.947247 28549 ts_tablet_manager.cc:1428] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260810 19:26:11.991492 28081 tablet.cc:2369] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:26:12.096652 28521 tablet.cc:2369] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:26:12.166011 28304 tablet.cc:2369] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:12.286798 28560 raft_consensus.cc:482] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:12.287168 28560 raft_consensus.cc:504] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:12.288888 28560 leader_election.cc:288] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd39b0771f04736bf4f930e85572137 (127.19.68.2:45629), 5e31feb5605b4b9585a2b60bfc8aa48c (127.19.68.3:36225)
I20260810 19:26:12.292275 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.289392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:35007 (local address 127.19.68.3:36225)
0810 19:26:12.289594 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.289601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.290468 (+ 867us) server_negotiation.cc:407] Connection header received
0810 19:26:12.290599 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.290605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.290656 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.290733 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.291669 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.291675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.291680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.291699 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.291730 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.291741 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.291848 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.291945 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.292009 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:26:12.292508 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.289486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:36225 (local address 127.19.68.1:35007)
0810 19:26:12.290361 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.290376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:12.290484 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:12.290772 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:12.290776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:12.290799 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:12.291480 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:12.291482 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:12.291495 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:12.291531 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:12.291821 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:12.291826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:12.291831 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:26:12.291981 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 19:26:12.292029 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":488,"threads_started":1}
I20260810 19:26:12.292539 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.289424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:50249 (local address 127.19.68.2:45629)
0810 19:26:12.289632 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.289638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.290421 (+ 783us) server_negotiation.cc:407] Connection header received
0810 19:26:12.290598 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.290603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.290661 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.290756 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.291886 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.291890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.291893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.291910 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.291942 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.291954 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.292077 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.292176 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.292254 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:26:12.293030 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.289372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:45629 (local address 127.19.68.1:50249)
0810 19:26:12.290296 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.290314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:12.290455 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:12.290795 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:12.290800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:12.290823 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:12.291651 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:26:12.291653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:12.291667 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:26:12.291724 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:12.292052 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:12.292062 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:12.292068 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:26:12.292228 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 19:26:12.292283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":458,"threads_started":1}
I20260810 19:26:12.293650 28438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a181084e981f499ea1c05f12da77cc31" candidate_uuid: "4c26baa558164def9c83dc987661f5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" is_pre_election: true
I20260810 19:26:12.294059 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a181084e981f499ea1c05f12da77cc31" candidate_uuid: "4c26baa558164def9c83dc987661f5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd39b0771f04736bf4f930e85572137" is_pre_election: true
I20260810 19:26:12.294297 28438 raft_consensus.cc:2454] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c26baa558164def9c83dc987661f5fb in term 0.
I20260810 19:26:12.294654 28212 raft_consensus.cc:2454] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c26baa558164def9c83dc987661f5fb in term 0.
I20260810 19:26:12.295014 27884 leader_election.cc:302] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c26baa558164def9c83dc987661f5fb, 5e31feb5605b4b9585a2b60bfc8aa48c; no voters:
I20260810 19:26:12.295434 28560 raft_consensus.cc:2781] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:12.295581 28560 raft_consensus.cc:482] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:12.295681 28560 raft_consensus.cc:3037] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:12.297847 28560 raft_consensus.cc:504] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:12.298368 28560 leader_election.cc:288] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [CANDIDATE]: Term 1 election: Requested vote from peers ddd39b0771f04736bf4f930e85572137 (127.19.68.2:45629), 5e31feb5605b4b9585a2b60bfc8aa48c (127.19.68.3:36225)
I20260810 19:26:12.298882 28438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a181084e981f499ea1c05f12da77cc31" candidate_uuid: "4c26baa558164def9c83dc987661f5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c"
I20260810 19:26:12.298949 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a181084e981f499ea1c05f12da77cc31" candidate_uuid: "4c26baa558164def9c83dc987661f5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd39b0771f04736bf4f930e85572137"
I20260810 19:26:12.299081 28438 raft_consensus.cc:3037] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:12.299130 28212 raft_consensus.cc:3037] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:12.301345 28212 raft_consensus.cc:2454] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c26baa558164def9c83dc987661f5fb in term 1.
I20260810 19:26:12.301621 28438 raft_consensus.cc:2454] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c26baa558164def9c83dc987661f5fb in term 1.
I20260810 19:26:12.301715 27882 leader_election.cc:302] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c26baa558164def9c83dc987661f5fb, ddd39b0771f04736bf4f930e85572137; no voters:
I20260810 19:26:12.302047 28560 raft_consensus.cc:2781] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:12.302541 28560 raft_consensus.cc:686] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 1 LEADER]: Becoming Leader. State: Replica: 4c26baa558164def9c83dc987661f5fb, State: Running, Role: LEADER
I20260810 19:26:12.303082 28560 consensus_queue.cc:237] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } }
I20260810 19:26:12.307835 27762 catalog_manager.cc:5495] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c26baa558164def9c83dc987661f5fb (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4c26baa558164def9c83dc987661f5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c26baa558164def9c83dc987661f5fb" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:12.354604 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.352756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.19.68.1:39893)
0810 19:26:12.352935 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.352941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.352960 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:26:12.353158 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.353162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.353251 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.353346 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.354131 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.354135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.354138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.354153 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.354185 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.354197 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.354304 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.354340 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.354386 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:12.362658 28212 raft_consensus.cc:1261] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 1 FOLLOWER]: Refusing update from remote peer 4c26baa558164def9c83dc987661f5fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:12.362663 28438 raft_consensus.cc:1261] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 1 FOLLOWER]: Refusing update from remote peer 4c26baa558164def9c83dc987661f5fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:12.363606 28567 consensus_queue.cc:1035] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd39b0771f04736bf4f930e85572137" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:12.363976 28560 consensus_queue.cc:1035] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e31feb5605b4b9585a2b60bfc8aa48c" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:12.396380 27763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:12.396665 27763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:12.400259 27763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 371aa7748f0541d899c4ed9011372c4e: Sending DeleteTablet for 3 replicas of tablet a181084e981f499ea1c05f12da77cc31
I20260810 19:26:12.401374 27971 tablet_service.cc:1514] Processing DeleteTablet for tablet a181084e981f499ea1c05f12da77cc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:12 UTC) from {username='slave'} at 127.0.0.1:52662
I20260810 19:26:12.401592 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet a181084e981f499ea1c05f12da77cc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:12 UTC) from {username='slave'} at 127.0.0.1:50410
I20260810 19:26:12.401643 28409 tablet_service.cc:1514] Processing DeleteTablet for tablet a181084e981f499ea1c05f12da77cc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:12 UTC) from {username='slave'} at 127.0.0.1:50782
I20260810 19:26:12.404984 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27860
I20260810 19:26:12.413708 28594 tablet_replica.cc:330] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: stopping tablet replica
I20260810 19:26:12.414237 28594 raft_consensus.cc:2229] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:12.415094 28594 raft_consensus.cc:2258] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:12.419689 28596 tablet_replica.cc:330] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: stopping tablet replica
I20260810 19:26:12.419986 28595 tablet_replica.cc:330] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: stopping tablet replica
I20260810 19:26:12.420161 28596 raft_consensus.cc:2229] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:12.420368 28595 raft_consensus.cc:2229] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:12.420713 28596 raft_consensus.cc:2258] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:12.420849 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.406497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.19.68.1:39893)
0810 19:26:12.406971 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.406978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.411491 (+ 4513us) server_negotiation.cc:407] Connection header received
0810 19:26:12.412375 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.412380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.412444 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.412522 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.419933 (+ 7411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.419939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.419943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.419965 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.420003 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.420016 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.420113 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.420497 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.420562 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260810 19:26:12.420830 28595 raft_consensus.cc:2258] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:12.421291 28594 ts_tablet_manager.cc:1905] T a181084e981f499ea1c05f12da77cc31 P 4c26baa558164def9c83dc987661f5fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:12.422086 28074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:12.424829 28596 ts_tablet_manager.cc:1905] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:12.425272 28595 ts_tablet_manager.cc:1905] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:12.431604 28596 ts_tablet_manager.cc:1918] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:12.431769 28596 log.cc:1196] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/a181084e981f499ea1c05f12da77cc31
I20260810 19:26:12.432281 28596 ts_tablet_manager.cc:1939] T a181084e981f499ea1c05f12da77cc31 P 5e31feb5605b4b9585a2b60bfc8aa48c: Deleting consensus metadata
I20260810 19:26:12.433326 28595 ts_tablet_manager.cc:1918] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:12.433506 28595 log.cc:1196] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/a181084e981f499ea1c05f12da77cc31
I20260810 19:26:12.434083 28595 ts_tablet_manager.cc:1939] T a181084e981f499ea1c05f12da77cc31 P ddd39b0771f04736bf4f930e85572137: Deleting consensus metadata
I20260810 19:26:12.434036 27745 catalog_manager.cc:4841] TS 5e31feb5605b4b9585a2b60bfc8aa48c (127.19.68.3:36225): tablet a181084e981f499ea1c05f12da77cc31 (table test-table [id=87d811edf3f54a34b15c6011ec7ebde8]) successfully deleted
I20260810 19:26:12.435926 27743 catalog_manager.cc:4841] TS ddd39b0771f04736bf4f930e85572137 (127.19.68.2:45629): tablet a181084e981f499ea1c05f12da77cc31 (table test-table [id=87d811edf3f54a34b15c6011ec7ebde8]) successfully deleted
I20260810 19:26:12.522955 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27860
I20260810 19:26:12.548344 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 28083
W20260810 19:26:12.548280 27738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:39893 (error 108)
I20260810 19:26:12.550448 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.549709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52692)
0810 19:26:12.550160 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.550270 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
W20260810 19:26:12.550709 27738 catalog_manager.cc:4584] TS 4c26baa558164def9c83dc987661f5fb (127.19.68.1:39893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a181084e981f499ea1c05f12da77cc31: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
I20260810 19:26:12.551432 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.549162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.19.68.2:45629)
0810 19:26:12.549362 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.549368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.549457 (+ 89us) server_negotiation.cc:407] Connection header received
0810 19:26:12.549645 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.549650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.549746 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.549834 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.550748 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.550754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.550758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.550789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.550829 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.550846 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.550975 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.551113 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.551163 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:26:12.552013 28292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:12.604506 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.603956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52702)
0810 19:26:12.604193 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.604317 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 19:26:12.605589 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.605195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52704)
0810 19:26:12.605342 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.605427 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 19:26:12.640405 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 28083
I20260810 19:26:12.666122 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.665584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52716)
0810 19:26:12.665828 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.665923 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:26:12.666227 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 28306
I20260810 19:26:12.667224 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.666765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52732)
0810 19:26:12.666964 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.667050 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 19:26:12.668830 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.666956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.19.68.3:36225)
0810 19:26:12.667113 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:12.667118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:12.667166 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:26:12.667381 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:12.667386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:12.667487 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:12.667571 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:12.668301 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:12.668306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:12.668310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:12.668324 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:12.668353 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:12.668364 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:12.668461 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:12.668513 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 19:26:12.668556 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:26:12.669461 28512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:12.748071 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.747581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52742)
0810 19:26:12.747813 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.747915 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 19:26:12.749234 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:12.748753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39893 (local address 127.0.0.1:52756)
0810 19:26:12.748958 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:12.749049 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 19:26:12.755156 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 28306
I20260810 19:26:12.782730 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27709
I20260810 19:26:12.783947 27812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:12.890846 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27709
2026-08-10T19:26:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2876 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 19:26:12.937994 19728 mini_oidc.cc:152] Starting JWKS server
I20260810 19:26:12.940774 19728 webserver.cc:460] Webserver started at https://127.0.0.1:42131/ using document root <none> and password file <none>
I20260810 19:26:12.941179 19728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:26:12.941808 19728 webserver.cc:460] Webserver started at http://127.0.0.1:38251/ using document root <none> and password file <none>
2026-08-10T19:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:12Z Disabled control of system clock
I20260810 19:26:12.974627 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:43785
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:43785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/testchainca.cert with env {}
W20260810 19:26:13.292003 28614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:13.292289 28614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:13.292420 28614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:13.299422 28614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:13.299516 28614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:13.299579 28614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:13.299655 28614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:13.299710 28614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:26:13.299772 28614 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:26:13.299825 28614 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42131/jwks.json
I20260810 19:26:13.309506 28614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:43785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:13.310722 28614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:13.311996 28614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:13.323239 28630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:13.323303 28628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:13.323330 28627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:13.324117 28614 server_base.cc:1034] running on GCE node
I20260810 19:26:13.324903 28614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:13.326313 28614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:13.327539 28614 hybrid_clock.cc:648] HybridClock initialized: now 1786389973327493 us; error 28 us; skew 500 ppm
I20260810 19:26:13.327894 28614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:13.330137 28614 webserver.cc:460] Webserver started at http://127.19.68.62:33037/ using document root <none> and password file <none>
I20260810 19:26:13.330783 28614 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:13.330899 28614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:13.331229 28614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:13.334234 28614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "e4ea8399411d4a5b82b5e1eae14f3b60"
format_stamp: "Formatted at 2026-08-10 19:26:13 on dist-test-slave-2j7r"
server_key: "dea4047c8519f46519a00c8b12d93f37"
server_key_iv: "a09ffcff46a9e418f6c180c79ae924e5"
server_key_version: "encryptionkey@0"
I20260810 19:26:13.335059 28614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "e4ea8399411d4a5b82b5e1eae14f3b60"
format_stamp: "Formatted at 2026-08-10 19:26:13 on dist-test-slave-2j7r"
server_key: "dea4047c8519f46519a00c8b12d93f37"
server_key_iv: "a09ffcff46a9e418f6c180c79ae924e5"
server_key_version: "encryptionkey@0"
I20260810 19:26:13.340184 28614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 19:26:13.343592 28640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:13.344852 28614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:13.345017 28614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "e4ea8399411d4a5b82b5e1eae14f3b60"
format_stamp: "Formatted at 2026-08-10 19:26:13 on dist-test-slave-2j7r"
server_key: "dea4047c8519f46519a00c8b12d93f37"
server_key_iv: "a09ffcff46a9e418f6c180c79ae924e5"
server_key_version: "encryptionkey@0"
I20260810 19:26:13.345255 28614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:13.359882 28614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:13.360594 28614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:13.360858 28614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:13.377172 28614 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:43785
I20260810 19:26:13.377208 28741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:43785 every 8 connection(s)
I20260810 19:26:13.378417 28614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:13.381860 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:13.382782 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 28614
I20260810 19:26:13.382975 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:13.383572 19728 external_mini_cluster.cc:1428] Setting key f48e2e56af33de4f338a26a138f3151d
I20260810 19:26:13.394830 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60: Bootstrap starting.
I20260810 19:26:13.396956 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:13.385367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.19.68.62:43785)
0810 19:26:13.386361 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:13.386372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:13.386416 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:26:13.387945 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:13.387968 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:13.388396 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:13.388705 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:13.389394 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.390623 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.392534 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.393272 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.393402 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:13.394687 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:13.394695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:13.394707 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:13.395266 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:13.395342 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:13.395364 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:13.395449 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:13.395709 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 19:26:13.396033 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":367,"threads_started":1}
I20260810 19:26:13.397989 28742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:13.399217 28742 log.cc:824] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:13.400696 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:26:13.402050 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60: No bootstrap required, opened a new log
I20260810 19:26:13.408984 28742 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } }
I20260810 19:26:13.409343 28742 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:13.409459 28742 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ea8399411d4a5b82b5e1eae14f3b60, State: Initialized, Role: FOLLOWER
I20260810 19:26:13.410001 28742 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } }
I20260810 19:26:13.410214 28742 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:13.410334 28742 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:13.410494 28742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:13.412493 28742 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } }
I20260810 19:26:13.412932 28742 leader_election.cc:302] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4ea8399411d4a5b82b5e1eae14f3b60; no voters:
I20260810 19:26:13.413461 28742 leader_election.cc:288] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:13.413601 28750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:13.414256 28750 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [term 1 LEADER]: Becoming Leader. State: Replica: e4ea8399411d4a5b82b5e1eae14f3b60, State: Running, Role: LEADER
I20260810 19:26:13.414812 28750 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } }
I20260810 19:26:13.415473 28742 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:13.417496 28755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ea8399411d4a5b82b5e1eae14f3b60. Latest consensus state: current_term: 1 leader_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } } }
I20260810 19:26:13.417518 28753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ea8399411d4a5b82b5e1eae14f3b60" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 43785 } } }
I20260810 19:26:13.417935 28753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:13.417932 28755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:13.420310 28764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:13.424780 28764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:13.432746 28764 catalog_manager.cc:1347] Generated new cluster ID: 8e33c6f41a5a40fdae2e40b471cc5b9b
I20260810 19:26:13.432844 28764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:13.440053 28764 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:13.441790 28764 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:13.453749 28764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4ea8399411d4a5b82b5e1eae14f3b60: Generated new TSK 0
I20260810 19:26:13.454721 28764 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 19:26:13.491631 28798 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 19:26:13.491830 28798 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:26:13.495249 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:13.470722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.19.68.62:43785)
0810 19:26:13.470940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:13.470946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:13.471481 (+ 535us) server_negotiation.cc:407] Connection header received
0810 19:26:13.471879 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:13.471885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:13.471985 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:13.472154 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:26:13.472766 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.473941 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.475431 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.476089 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.476214 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:13.476726 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:26:13.494539 (+ 17813us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 19:26:13.494928 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 19:26:13.495388 28748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 19:26:13.497120 28796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:13.470613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:43785 (local address 127.0.0.1:47842)
0810 19:26:13.471368 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:13.471389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:13.471563 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:13.472179 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:13.472183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:13.472194 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:26:13.472604 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:13.472615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.474106 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.474111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:13.475287 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:13.475300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:13.476250 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:13.476254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:13.476375 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:13.476448 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:26:13.494841 (+ 18393us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 19:26:13.495119 (+ 278us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.68.62:43785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":391,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.68.62:43785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 19:26:13.499246 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 28614
I20260810 19:26:13.500305 28735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:13.592123 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 28614
2026-08-10T19:26:13Z chronyd exiting
I20260810 19:26:13.647274 19728 test_util.cc:175] -----------------------------------------------
I20260810 19:26:13.647420 19728 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389929706493-19728-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 19:26:13.652136 19728 mini_oidc.cc:152] Starting JWKS server
I20260810 19:26:13.653985 19728 webserver.cc:460] Webserver started at https://127.0.0.1:38283/ using document root <none> and password file <none>
I20260810 19:26:13.654212 19728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:26:13.654847 19728 webserver.cc:460] Webserver started at http://127.0.0.1:35553/ using document root <none> and password file <none>
2026-08-10T19:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:13Z Disabled control of system clock
I20260810 19:26:13.687966 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38957
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:38957
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38283/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/testchainca.cert with env {}
W20260810 19:26:14.014147 28809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:14.014441 28809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:14.014581 28809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:14.021715 28809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:14.021826 28809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:14.021890 28809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:14.021967 28809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:14.022037 28809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:26:14.022130 28809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:26:14.022182 28809 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38283/jwks.json
I20260810 19:26:14.031905 28809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:38957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38283/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:14.033159 28809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:14.034482 28809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:14.045920 28820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:14.046000 28825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:14.045975 28821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:14.046808 28809 server_base.cc:1034] running on GCE node
I20260810 19:26:14.047542 28809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:14.048890 28809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:14.050094 28809 hybrid_clock.cc:648] HybridClock initialized: now 1786389974050048 us; error 35 us; skew 500 ppm
I20260810 19:26:14.050436 28809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:14.060182 28809 webserver.cc:460] Webserver started at http://127.19.68.62:38663/ using document root <none> and password file <none>
I20260810 19:26:14.060935 28809 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:14.061053 28809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:14.061434 28809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:14.064337 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "0d35a7f0e07e40828a7c80bf864df9e7"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "14e34bc040e085b41547e4dc17a65c6e"
server_key_iv: "b3b200fa4504ee6cb412afeb5810383c"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.065181 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "0d35a7f0e07e40828a7c80bf864df9e7"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "14e34bc040e085b41547e4dc17a65c6e"
server_key_iv: "b3b200fa4504ee6cb412afeb5810383c"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.070225 28809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:26:14.073650 28838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:14.074962 28809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:26:14.075127 28809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "0d35a7f0e07e40828a7c80bf864df9e7"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "14e34bc040e085b41547e4dc17a65c6e"
server_key_iv: "b3b200fa4504ee6cb412afeb5810383c"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.075325 28809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:14.096004 28809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:14.096818 28809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:14.097157 28809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:14.114446 28809 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:38957
I20260810 19:26:14.114454 28921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:38957 every 8 connection(s)
I20260810 19:26:14.115727 28809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:14.119355 28922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:14.126207 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 28809
I20260810 19:26:14.126406 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:14.127149 19728 external_mini_cluster.cc:1428] Setting key 3ec961ea6acaaf9e3f6dcef63d8c7644
I20260810 19:26:14.129786 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7: Bootstrap starting.
I20260810 19:26:14.132884 28922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:14.133984 28922 log.cc:824] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:14.136595 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7: No bootstrap required, opened a new log
I20260810 19:26:14.139901 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:14.129195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.19.68.62:38957)
0810 19:26:14.130183 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:14.130193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:14.130240 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:26:14.131906 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:14.131927 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:14.132343 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:14.132675 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:14.133250 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.134532 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.135777 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.136435 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.136605 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.137713 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:14.137721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:14.137734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:14.138259 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:14.138339 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:14.138361 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:14.138474 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:14.138684 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 19:26:14.138978 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":332,"threads_started":1}
I20260810 19:26:14.143404 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:26:14.144240 28922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:26:14.144562 28922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:14.144647 28922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d35a7f0e07e40828a7c80bf864df9e7, State: Initialized, Role: FOLLOWER
I20260810 19:26:14.145226 28922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:26:14.145462 28922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:14.145617 28922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:14.145787 28922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:14.148123 28922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:26:14.148563 28922 leader_election.cc:302] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d35a7f0e07e40828a7c80bf864df9e7; no voters:
I20260810 19:26:14.149188 28922 leader_election.cc:288] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:14.149315 28933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:14.150101 28933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [term 1 LEADER]: Becoming Leader. State: Replica: 0d35a7f0e07e40828a7c80bf864df9e7, State: Running, Role: LEADER
I20260810 19:26:14.150566 28933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:26:14.151094 28922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:14.153173 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } } }
I20260810 19:26:14.153538 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:14.154383 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d35a7f0e07e40828a7c80bf864df9e7. Latest consensus state: current_term: 1 leader_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35a7f0e07e40828a7c80bf864df9e7" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } } }
I20260810 19:26:14.154695 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:14.156308 28942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:14.161453 28942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:14.170121 28942 catalog_manager.cc:1347] Generated new cluster ID: f7b2952ed12c4577a65c0a161ceb5859
I20260810 19:26:14.170228 28942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:14.192063 28942 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:14.193354 28942 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:14.201563 28942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d35a7f0e07e40828a7c80bf864df9e7: Generated new TSK 0
I20260810 19:26:14.202373 28942 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 19:26:14.239081 28972 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 19:26:14.239202 28972 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:26:14.240520 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:14.220653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.19.68.62:38957)
0810 19:26:14.220835 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:14.220841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:14.221217 (+ 376us) server_negotiation.cc:407] Connection header received
0810 19:26:14.221436 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:14.221446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:14.221541 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:14.221713 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:26:14.222127 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.223254 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.224601 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.225334 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.225472 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.225924 (+ 452us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:26:14.239773 (+ 13849us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 19:26:14.240206 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 19:26:14.240659 28928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 19:26:14.240715 28971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:14.220471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38957 (local address 127.0.0.1:44480)
0810 19:26:14.221111 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:14.221131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:14.221260 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:14.221713 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:14.221716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:14.221725 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:26:14.221968 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:14.221977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.223375 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.223378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:14.224463 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:14.224475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.225487 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.225491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:14.225619 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.225653 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:26:14.240061 (+ 14408us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 19:26:14.240318 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.68.62:38957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":319,"threads_started":1}
I20260810 19:26:14.242259 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 28809
I20260810 19:26:14.243326 28914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:14.334766 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 28809
2026-08-10T19:26:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 19:26:14.398851 19728 mini_oidc.cc:152] Starting JWKS server
I20260810 19:26:14.400746 19728 webserver.cc:460] Webserver started at https://127.0.0.1:35007/ using document root <none> and password file <none>
I20260810 19:26:14.400990 19728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:26:14.401624 19728 webserver.cc:460] Webserver started at http://127.0.0.1:44911/ using document root <none> and password file <none>
2026-08-10T19:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:14Z Disabled control of system clock
I20260810 19:26:14.436002 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:41761
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:41761
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 19:26:14.757915 28989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:14.758194 28989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:14.758338 28989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:14.765267 28989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:14.765379 28989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:14.765448 28989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:14.765525 28989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:14.765584 28989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:26:14.765666 28989 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:26:14.765719 28989 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35007/jwks.json
I20260810 19:26:14.774678 28989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:41761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:14.775957 28989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:14.777720 28989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:14.789001 28997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:14.789065 29001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:14.788964 28998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:14.789899 28989 server_base.cc:1034] running on GCE node
I20260810 19:26:14.790581 28989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:14.791842 28989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:14.793057 28989 hybrid_clock.cc:648] HybridClock initialized: now 1786389974793009 us; error 47 us; skew 500 ppm
I20260810 19:26:14.793406 28989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:14.795583 28989 webserver.cc:460] Webserver started at http://127.19.68.62:37141/ using document root <none> and password file <none>
I20260810 19:26:14.796229 28989 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:14.796324 28989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:14.796631 28989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:14.799428 28989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "df132f0a0e2b47968ca2f5d530605158"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "62d9b36d826214cbe3b6dcb104b4ff42"
server_key_iv: "55278b6c031e4edeeb2db645da9277c9"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.800179 28989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "df132f0a0e2b47968ca2f5d530605158"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "62d9b36d826214cbe3b6dcb104b4ff42"
server_key_iv: "55278b6c031e4edeeb2db645da9277c9"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.805068 28989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:26:14.808633 29009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:14.809933 28989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:26:14.810091 28989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "df132f0a0e2b47968ca2f5d530605158"
format_stamp: "Formatted at 2026-08-10 19:26:14 on dist-test-slave-2j7r"
server_key: "62d9b36d826214cbe3b6dcb104b4ff42"
server_key_iv: "55278b6c031e4edeeb2db645da9277c9"
server_key_version: "encryptionkey@0"
I20260810 19:26:14.810287 28989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:14.842504 28989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:14.843851 28989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:14.844163 28989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:14.861272 28989 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:41761
I20260810 19:26:14.861295 29107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:41761 every 8 connection(s)
I20260810 19:26:14.863055 28989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:14.865688 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 28989
I20260810 19:26:14.865900 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:14.866743 19728 external_mini_cluster.cc:1428] Setting key 48f39947a8483ee1c99cf69b2e9ed568
I20260810 19:26:14.867801 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:14.879915 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:14.868704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.19.68.62:41761)
0810 19:26:14.869811 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:14.869829 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:14.869877 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:26:14.871576 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:14.871597 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:14.872067 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:14.872428 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:14.872996 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.874405 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.875702 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.876358 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.876522 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.877721 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:14.877729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:14.877741 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:14.878291 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:14.878365 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:14.878389 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:14.878494 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:14.878701 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 19:26:14.879016 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":381,"threads_started":1}
I20260810 19:26:14.880060 29108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158: Bootstrap starting.
I20260810 19:26:14.883559 29108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:14.883838 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:26:14.884557 29108 log.cc:824] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:14.887497 29108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158: No bootstrap required, opened a new log
I20260810 19:26:14.894768 29108 raft_consensus.cc:348] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } }
I20260810 19:26:14.895179 29108 raft_consensus.cc:374] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:14.895284 29108 raft_consensus.cc:729] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df132f0a0e2b47968ca2f5d530605158, State: Initialized, Role: FOLLOWER
I20260810 19:26:14.895807 29108 consensus_queue.cc:260] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } }
I20260810 19:26:14.896015 29108 raft_consensus.cc:388] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:14.896109 29108 raft_consensus.cc:482] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:14.896231 29108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:14.898273 29108 raft_consensus.cc:504] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } }
I20260810 19:26:14.898690 29108 leader_election.cc:302] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df132f0a0e2b47968ca2f5d530605158; no voters:
I20260810 19:26:14.899185 29108 leader_election.cc:288] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:14.899310 29113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:14.899931 29113 raft_consensus.cc:686] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [term 1 LEADER]: Becoming Leader. State: Replica: df132f0a0e2b47968ca2f5d530605158, State: Running, Role: LEADER
I20260810 19:26:14.900444 29113 consensus_queue.cc:237] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } }
I20260810 19:26:14.901288 29108 sys_catalog.cc:564] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:14.903481 29115 sys_catalog.cc:455] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df132f0a0e2b47968ca2f5d530605158. Latest consensus state: current_term: 1 leader_uuid: "df132f0a0e2b47968ca2f5d530605158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } } }
I20260810 19:26:14.903515 29114 sys_catalog.cc:455] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df132f0a0e2b47968ca2f5d530605158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df132f0a0e2b47968ca2f5d530605158" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41761 } } }
I20260810 19:26:14.903887 29115 sys_catalog.cc:458] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:14.903898 29114 sys_catalog.cc:458] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:14.905623 29124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:14.910502 29124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:14.919211 29124 catalog_manager.cc:1347] Generated new cluster ID: 064a10c8fd374bf8b2abee1ac93affb6
I20260810 19:26:14.919308 29124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:14.932763 29124 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:14.934047 29124 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:14.942869 29124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df132f0a0e2b47968ca2f5d530605158: Generated new TSK 0
I20260810 19:26:14.943614 29124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:26:14.965440 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:14.957202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.19.68.62:41761)
0810 19:26:14.957355 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:14.957359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:14.957699 (+ 340us) server_negotiation.cc:407] Connection header received
0810 19:26:14.957910 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:14.957916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:14.958008 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:14.958177 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:26:14.958638 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.959753 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.961075 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.961793 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.961926 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.962373 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:26:14.964763 (+ 2390us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 19:26:14.965123 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 19:26:14.965580 29111 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 19:26:14.967104 29153 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:14.957017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:41761 (local address 127.0.0.1:39874)
0810 19:26:14.957591 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:14.957610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:14.957727 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:14.958191 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:14.958194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:14.958205 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:26:14.958486 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:14.958495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.959878 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.959881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:14.960935 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:14.960946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:14.961966 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:14.961971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:14.962097 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:14.962129 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:26:14.964972 (+ 2843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 19:26:14.966679 (+ 1707us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.68.62:41761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1426,"thread_start_us":325,"threads_started":1}
I20260810 19:26:14.968104 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 28989
I20260810 19:26:14.969240 29098 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 19:26:14.976660 29154 webserver.cc:533] Webserver: sslize failed:
E20260810 19:26:14.976754 29154 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:26:15.060413 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 28989
2026-08-10T19:26:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T19:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:15Z Disabled control of system clock
I20260810 19:26:15.158213 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44629
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:44629
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/testchainca.cert with env {}
W20260810 19:26:15.476526 29164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:15.476855 29164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:15.477001 29164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:15.484292 29164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:15.484426 29164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:15.484503 29164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:15.484591 29164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:15.484656 29164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:26:15.484752 29164 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:26:15.484812 29164 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 19:26:15.495026 29164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:44629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:15.496352 29164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:15.497762 29164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:15.509898 29174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:15.509827 29178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:15.510022 29180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:15.510483 29164 server_base.cc:1034] running on GCE node
I20260810 19:26:15.511181 29164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:15.512476 29164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:15.513654 29164 hybrid_clock.cc:648] HybridClock initialized: now 1786389975513608 us; error 43 us; skew 500 ppm
I20260810 19:26:15.513972 29164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:26:15.516413 29164 webserver.cc:460] Webserver started at http://127.19.68.62:44491/ using document root <none> and password file <none>
I20260810 19:26:15.517058 29164 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:15.517196 29164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:15.517499 29164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:15.520232 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "472c3927604b4ac7bfe9989332a67351"
format_stamp: "Formatted at 2026-08-10 19:26:15 on dist-test-slave-2j7r"
server_key: "df81fdc6111d63927ad17d6d5314b028"
server_key_iv: "d0efdb8e434f2f11dd9e666d062a85ab"
server_key_version: "encryptionkey@0"
I20260810 19:26:15.521008 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "472c3927604b4ac7bfe9989332a67351"
format_stamp: "Formatted at 2026-08-10 19:26:15 on dist-test-slave-2j7r"
server_key: "df81fdc6111d63927ad17d6d5314b028"
server_key_iv: "d0efdb8e434f2f11dd9e666d062a85ab"
server_key_version: "encryptionkey@0"
I20260810 19:26:15.525856 29164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:26:15.529032 29189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:15.530299 29164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:26:15.530460 29164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "472c3927604b4ac7bfe9989332a67351"
format_stamp: "Formatted at 2026-08-10 19:26:15 on dist-test-slave-2j7r"
server_key: "df81fdc6111d63927ad17d6d5314b028"
server_key_iv: "d0efdb8e434f2f11dd9e666d062a85ab"
server_key_version: "encryptionkey@0"
I20260810 19:26:15.530650 29164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:15.553345 29164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:15.554142 29164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:15.554415 29164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:15.571106 29164 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:44629
I20260810 19:26:15.571130 29290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:44629 every 8 connection(s)
I20260810 19:26:15.572379 29164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:15.575910 29292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:15.577594 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 29164
I20260810 19:26:15.577790 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:15.578627 19728 external_mini_cluster.cc:1428] Setting key f5abd7ec3b3749b850fb5747793e9a02
I20260810 19:26:15.587260 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351: Bootstrap starting.
I20260810 19:26:15.590253 29292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:15.591202 29292 log.cc:824] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:15.593912 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351: No bootstrap required, opened a new log
I20260810 19:26:15.595759 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:15.580617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.19.68.62:44629)
0810 19:26:15.581593 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:15.581603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:15.581645 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:26:15.583441 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:15.583465 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:15.583927 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:15.584371 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:15.584960 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.586364 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.587714 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.588403 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.588610 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:15.593185 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:15.593193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:15.593205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:26:15.593718 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:15.593792 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:26:15.593812 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:15.593921 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:15.594202 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 19:26:15.594532 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":319,"threads_started":1}
I20260810 19:26:15.599392 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:26:15.602753 29292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } }
I20260810 19:26:15.603087 29292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:15.603185 29292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472c3927604b4ac7bfe9989332a67351, State: Initialized, Role: FOLLOWER
I20260810 19:26:15.603725 29292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } }
I20260810 19:26:15.603935 29292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:15.604054 29292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:15.604198 29292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:15.606309 29292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } }
I20260810 19:26:15.606761 29292 leader_election.cc:302] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472c3927604b4ac7bfe9989332a67351; no voters:
I20260810 19:26:15.607329 29292 leader_election.cc:288] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:15.607462 29301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:15.608140 29301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [term 1 LEADER]: Becoming Leader. State: Replica: 472c3927604b4ac7bfe9989332a67351, State: Running, Role: LEADER
I20260810 19:26:15.608639 29301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } }
I20260810 19:26:15.609531 29292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:15.611411 29304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472c3927604b4ac7bfe9989332a67351. Latest consensus state: current_term: 1 leader_uuid: "472c3927604b4ac7bfe9989332a67351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } } }
I20260810 19:26:15.611460 29302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472c3927604b4ac7bfe9989332a67351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c3927604b4ac7bfe9989332a67351" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44629 } } }
I20260810 19:26:15.611776 29304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:15.611817 29302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351 [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:15.613950 29309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:15.618597 29309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:15.627138 29309 catalog_manager.cc:1347] Generated new cluster ID: 71d047ef021d4e89b1062bf15b85a7f7
I20260810 19:26:15.627292 29309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:15.635699 29309 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:15.637481 29309 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:15.646692 29309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472c3927604b4ac7bfe9989332a67351: Generated new TSK 0
I20260810 19:26:15.647768 29309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:26:16.506640 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:15.661946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.19.68.62:44629)
0810 19:26:15.662144 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:15.662150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:15.662548 (+ 398us) server_negotiation.cc:407] Connection header received
0810 19:26:15.662783 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:15.662786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:15.662852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:15.662980 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:26:15.663569 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.664269 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.666429 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.666889 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.666979 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:15.667552 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:26:16.505810 (+838258us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 19:26:16.506387 (+ 577us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59814: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 19:26:16.506764 29297 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59814: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 19:26:16.506829 29335 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:15.661731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44629 (local address 127.0.0.1:59814)
0810 19:26:15.662436 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:15.662458 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:15.662623 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:15.663026 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:15.663032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:15.663056 (+ 24us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:26:15.663406 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:15.663421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.664423 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.664430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:15.666279 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:15.666298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:15.667069 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:15.667077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:15.667274 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:15.667326 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:26:16.506192 (+838866us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 19:26:16.506496 (+ 304us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.68.62:44629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":379,"threads_started":1}
I20260810 19:26:16.507838 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 29164
I20260810 19:26:16.509014 29281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:16.611629 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 29164
2026-08-10T19:26:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1533 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (413 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:17 dist-test-slave-2j7r krb5kdc[29369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:17 dist-test-slave-2j7r krb5kdc[29369](info): set up 2 sockets
Aug 10 19:26:17 dist-test-slave-2j7r krb5kdc[29369](info): commencing operation
krb5kdc: starting...
W20260810 19:26:18.692773 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:18 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:26:18.928027 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37687
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:19.253523 29391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:19.253813 29391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:19.253926 29391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:19.261143 29391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:19.261250 29391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:19.261300 29391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:19.261376 29391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:19.261430 29391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:19.271342 29391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:19.272712 29391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:19.274049 29391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:19.285740 29399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:19.285673 29403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:19.285648 29400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:19.286375 29391 server_base.cc:1034] running on GCE node
I20260810 19:26:19.287127 29391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:19.288506 29391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:19.289716 29391 hybrid_clock.cc:648] HybridClock initialized: now 1786389979289657 us; error 44 us; skew 500 ppm
I20260810 19:26:19.290076 29391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:19 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389979, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:19.296108 29391 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:26:19.297619 29391 webserver.cc:460] Webserver started at http://127.19.68.62:37409/ using document root <none> and password file <none>
I20260810 19:26:19.298303 29391 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:19.298409 29391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:19.298682 29391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:19.301862 29391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "7ae1937ddae04528a29f3b558e40e94d"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "6f035d1831698dd04305e52bc8105bce"
server_key_iv: "f3ac8e078f77b1b329f655b29d14429d"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.302667 29391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "7ae1937ddae04528a29f3b558e40e94d"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "6f035d1831698dd04305e52bc8105bce"
server_key_iv: "f3ac8e078f77b1b329f655b29d14429d"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.308228 29391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:26:19.311982 29412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:19.313540 29391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:26:19.313717 29391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "7ae1937ddae04528a29f3b558e40e94d"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "6f035d1831698dd04305e52bc8105bce"
server_key_iv: "f3ac8e078f77b1b329f655b29d14429d"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.313917 29391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:19.327610 29391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:19.331867 29391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:19.332131 29391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:19.348670 29391 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:37687
I20260810 19:26:19.348693 29495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:37687 every 8 connection(s)
I20260810 19:26:19.349990 29391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:19.351017 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 29391
I20260810 19:26:19.351228 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:19.351873 19728 external_mini_cluster.cc:1428] Setting key 452977321b43a7fa692fcf01e23a71e4
I20260810 19:26:19.354168 29496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:26:19 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:19.366647 29496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d: Bootstrap starting.
I20260810 19:26:19.370468 29496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:19.371632 29496 log.cc:824] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:19.374940 29496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d: No bootstrap required, opened a new log
I20260810 19:26:19.375442 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:19.353782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.19.68.62:37687)
0810 19:26:19.354549 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:19.354564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:19.354613 (+ 49us) server_negotiation.cc:407] Connection header received
0810 19:26:19.356949 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:19.356975 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:19.357475 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:19.357893 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:19.359190 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.361192 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.362424 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.363401 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.363576 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:19.367302 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:19.367312 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:19.367328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:19.367376 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:19.371042 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:19.371728 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:19.371736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:19.371743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:19.371970 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:19.372473 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:19.372476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:19.372479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:19.373042 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:19.373345 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:19.373759 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 19:26:19.374176 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
I20260810 19:26:19.382886 29496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } }
I20260810 19:26:19.383206 29496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:19.383282 29496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae1937ddae04528a29f3b558e40e94d, State: Initialized, Role: FOLLOWER
I20260810 19:26:19.383742 29496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } }
I20260810 19:26:19.383982 29496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:19.384119 29496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:19.384317 29496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:19.386667 29496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } }
I20260810 19:26:19.387125 29496 leader_election.cc:302] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ae1937ddae04528a29f3b558e40e94d; no voters:
I20260810 19:26:19.387835 29496 leader_election.cc:288] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:19.387948 29503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:19.389070 29503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [term 1 LEADER]: Becoming Leader. State: Replica: 7ae1937ddae04528a29f3b558e40e94d, State: Running, Role: LEADER
I20260810 19:26:19.389642 29503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } }
I20260810 19:26:19.390081 29496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:19.392256 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ae1937ddae04528a29f3b558e40e94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } } }
I20260810 19:26:19.392549 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:19.394037 29516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:19.394029 29507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ae1937ddae04528a29f3b558e40e94d. Latest consensus state: current_term: 1 leader_uuid: "7ae1937ddae04528a29f3b558e40e94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae1937ddae04528a29f3b558e40e94d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37687 } } }
I20260810 19:26:19.394369 29507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:19.399256 29516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:19.408038 29516 catalog_manager.cc:1347] Generated new cluster ID: 93702a1ae2c94327a4d9c1a2d1b031fc
I20260810 19:26:19.408138 29516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:19.432643 29516 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:19.434000 29516 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:19.447269 29516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ae1937ddae04528a29f3b558e40e94d: Generated new TSK 0
I20260810 19:26:19.448118 29516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:26:19.512207 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:19.839947 29538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:19.840222 29538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:19.840325 29538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:19.848093 29538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:19.848199 29538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:19.848320 29538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:19.860814 29538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:19.862579 29538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:19.864315 29538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:19.879894 29547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:19.881768 29548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:19.883896 29553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:19.885560 29538 server_base.cc:1034] running on GCE node
I20260810 19:26:19.886314 29538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:19.887562 29538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:19.888777 29538 hybrid_clock.cc:648] HybridClock initialized: now 1786389979888715 us; error 53 us; skew 500 ppm
I20260810 19:26:19.889160 29538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:19 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389979, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:19.893466 29538 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:26:19.896633 29538 webserver.cc:460] Webserver started at http://127.19.68.1:35743/ using document root <none> and password file <none>
I20260810 19:26:19.897572 29538 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:19.897697 29538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:19.897969 29538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:19.900851 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "7245768429fb4363bb639c3e7d1bfff8"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "2e2cd97070fc9c86517f2ce791e0adc4"
server_key_iv: "f854213eda8b116364f884a990e0768c"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.901648 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "7245768429fb4363bb639c3e7d1bfff8"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "2e2cd97070fc9c86517f2ce791e0adc4"
server_key_iv: "f854213eda8b116364f884a990e0768c"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.906777 29538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:26:19.910104 29563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:19.911377 29538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:26:19.911545 29538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "7245768429fb4363bb639c3e7d1bfff8"
format_stamp: "Formatted at 2026-08-10 19:26:19 on dist-test-slave-2j7r"
server_key: "2e2cd97070fc9c86517f2ce791e0adc4"
server_key_iv: "f854213eda8b116364f884a990e0768c"
server_key_version: "encryptionkey@0"
I20260810 19:26:19.911717 29538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:19.940423 29538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:19.944756 29538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:19.945020 29538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:19.946014 29538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:19.947491 29538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:19.947566 29538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:19.947688 29538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:19.947746 29538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:19.977615 29538 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:38583
I20260810 19:26:19.977685 29774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:38583 every 8 connection(s)
I20260810 19:26:19.978860 29538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:19.988551 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 29538
I20260810 19:26:19.988729 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:19.989421 19728 external_mini_cluster.cc:1428] Setting key 0406f35a5ad6b6ac7b5506cdbbca87ee
Aug 10 19:26:19 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389979, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:19.998603 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:19.981268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:50695 (local address 127.19.68.62:37687)
0810 19:26:19.981712 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:19.981716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:19.982565 (+ 849us) server_negotiation.cc:407] Connection header received
0810 19:26:19.984391 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:19.984397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:19.984488 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:19.984601 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:19.986116 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.986850 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.988401 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.988867 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.988986 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:19.993134 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:19.993141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:19.993146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:19.993183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:19.995590 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:19.996434 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:19.996437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:19.996440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:19.996559 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:19.996997 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:19.997000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:19.997002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:19.997296 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:19.997406 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:19.998169 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 19:26:19.998342 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260810 19:26:19.999797 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:19.981689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37687 (local address 127.19.68.1:50695)
0810 19:26:19.982399 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:19.982438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:19.984116 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:19.984833 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:19.984841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:19.985386 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:19.985934 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:19.985966 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.987000 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.987004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:19.988250 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:19.988274 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:19.989111 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:19.989119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:19.989347 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:19.990020 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:19.990050 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:19.992845 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:19.995714 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:19.995722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:19.995735 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:19.996325 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:19.996657 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:19.996661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:19.996664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:19.996887 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:19.997413 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:19.997420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:19.997797 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 19:26:19.998083 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 19:26:19.998562 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260810 19:26:20.001322 29776 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37687
I20260810 19:26:20.001611 29776 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:20.002305 29776 heartbeater.cc:511] Master 127.19.68.62:37687 requested a full tablet report, sending...
I20260810 19:26:20.004545 29438 ts_manager.cc:194] Registered new tserver with Master: 7245768429fb4363bb639c3e7d1bfff8 (127.19.68.1:38583)
I20260810 19:26:20.006531 29438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:50695
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:26:20.042552 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:20.365700 29789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:20.365940 29789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:20.366060 29789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:20.372938 29789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:20.373028 29789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:20.373175 29789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:20.382491 29789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:20.383657 29789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:20.384897 29789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:20.397557 29799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:20.397575 29802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:20.398152 29789 server_base.cc:1034] running on GCE node
W20260810 19:26:20.398258 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:20.398882 29789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:20.399773 29789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:20.400978 29789 hybrid_clock.cc:648] HybridClock initialized: now 1786389980400925 us; error 44 us; skew 500 ppm
I20260810 19:26:20.401293 29789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:20 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:20.404258 29789 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:26:20.405704 29789 webserver.cc:460] Webserver started at http://127.19.68.2:34983/ using document root <none> and password file <none>
I20260810 19:26:20.406353 29789 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:20.406471 29789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:20.406760 29789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:20.409700 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "51b60e63877b426c94055362a94a7fd7"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "97da4b32acb91a5cf21fa83dd8784e17"
server_key_iv: "69fdd5bf77f9bb4e5f676de4f30b9aa1"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.410446 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "51b60e63877b426c94055362a94a7fd7"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "97da4b32acb91a5cf21fa83dd8784e17"
server_key_iv: "69fdd5bf77f9bb4e5f676de4f30b9aa1"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.415452 29789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 19:26:20.418991 29817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.420274 29789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:20.420428 29789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "51b60e63877b426c94055362a94a7fd7"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "97da4b32acb91a5cf21fa83dd8784e17"
server_key_iv: "69fdd5bf77f9bb4e5f676de4f30b9aa1"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.420616 29789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:20.437168 29789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:20.441157 29789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:20.441418 29789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:20.442435 29789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:20.443815 29789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:20.443904 29789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.444031 29789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:20.444085 29789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.474062 29789 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:41635
I20260810 19:26:20.474088 30020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:41635 every 8 connection(s)
I20260810 19:26:20.475286 29789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:20.483697 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 29789
I20260810 19:26:20.483875 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:20.484475 19728 external_mini_cluster.cc:1428] Setting key bdf0611886933076d8358217f252643d
Aug 10 19:26:20 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:20.495702 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:20.477715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:48933 (local address 127.19.68.62:37687)
0810 19:26:20.477881 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:20.477885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:20.479019 (+ 1134us) server_negotiation.cc:407] Connection header received
0810 19:26:20.480748 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:20.480752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:20.480825 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:20.480913 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:20.482373 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.483132 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.485315 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.485964 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.486081 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:20.490000 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:20.490005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:20.490009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:20.490044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:20.492619 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:20.493462 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:20.493466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:20.493468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:20.493597 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:20.494032 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:20.494034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:20.494036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:20.494324 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:20.494440 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:20.495221 (+ 781us) server_negotiation.cc:299] Negotiation successful
0810 19:26:20.495459 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:26:20.496977 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:20.478125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37687 (local address 127.19.68.2:48933)
0810 19:26:20.478826 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:20.478873 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:20.480474 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:20.481147 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:20.481155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:20.481643 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:20.482199 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:20.482215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.483296 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.483302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:20.485179 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:20.485192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.486160 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.486165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:20.486416 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:20.487066 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:20.487095 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:20.489743 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:20.492747 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:20.492755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:20.492772 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:20.493340 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:20.493693 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:20.493697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:20.493700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:20.493940 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:20.494435 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:20.494442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:20.494836 (+ 394us) client_negotiation.cc:769] Sending connection context
0810 19:26:20.495136 (+ 300us) client_negotiation.cc:240] Negotiation successful
0810 19:26:20.495653 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":267,"threads_started":1}
I20260810 19:26:20.498418 30021 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37687
I20260810 19:26:20.498749 30021 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:20.499577 30021 heartbeater.cc:511] Master 127.19.68.62:37687 requested a full tablet report, sending...
I20260810 19:26:20.501144 29438 ts_manager.cc:194] Registered new tserver with Master: 51b60e63877b426c94055362a94a7fd7 (127.19.68.2:41635)
I20260810 19:26:20.501952 29438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:48933
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:26:20.540796 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:20.864984 30034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:20.865336 30034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:20.865492 30034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:20.873128 30034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:20.873240 30034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:20.873404 30034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:20.882910 30034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:20.884130 30034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:20.885434 30034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:20.897936 30042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:20.897889 30047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:20.897971 30043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:20.898761 30034 server_base.cc:1034] running on GCE node
I20260810 19:26:20.899339 30034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:20.900214 30034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:20.901419 30034 hybrid_clock.cc:648] HybridClock initialized: now 1786389980901365 us; error 38 us; skew 500 ppm
I20260810 19:26:20.901700 30034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:20 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:20.904642 30034 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:26:20.905998 30034 webserver.cc:460] Webserver started at http://127.19.68.3:39189/ using document root <none> and password file <none>
I20260810 19:26:20.906634 30034 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:20.906749 30034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:20.907037 30034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:20.909860 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "cb5c8b1b0c8a4154b212babc942eee6e"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "3f751c6045a02a82936c5620d16d4977"
server_key_iv: "ed6a3f2d130370a0e3d886a51354927b"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.910609 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "cb5c8b1b0c8a4154b212babc942eee6e"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "3f751c6045a02a82936c5620d16d4977"
server_key_iv: "ed6a3f2d130370a0e3d886a51354927b"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.915501 30034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:20.919127 30061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.920398 30034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:20.920543 30034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "cb5c8b1b0c8a4154b212babc942eee6e"
format_stamp: "Formatted at 2026-08-10 19:26:20 on dist-test-slave-2j7r"
server_key: "3f751c6045a02a82936c5620d16d4977"
server_key_iv: "ed6a3f2d130370a0e3d886a51354927b"
server_key_version: "encryptionkey@0"
I20260810 19:26:20.920717 30034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:20.937712 30034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:20.941915 30034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:20.942184 30034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:20.943151 30034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:20.944473 30034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:20.944543 30034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.944660 30034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:20.944710 30034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:20.973902 30034 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42903
I20260810 19:26:20.973956 30259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42903 every 8 connection(s)
I20260810 19:26:20.975159 30034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:20.984804 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 30034
I20260810 19:26:20.984995 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:20.985482 19728 external_mini_cluster.cc:1428] Setting key 155f364a6f8a00a8b9467c0afb47635d
Aug 10 19:26:20 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:20.995065 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:20.977726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:41445 (local address 127.19.68.62:37687)
0810 19:26:20.977898 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:20.977904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:20.979283 (+ 1379us) server_negotiation.cc:407] Connection header received
0810 19:26:20.981034 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:20.981040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:20.981151 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:20.981242 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:20.982696 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.983450 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.985195 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.985632 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.985732 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:20.989420 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:20.989425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:20.989429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:20.989462 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:20.992244 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:20.992902 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:20.992908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:20.992910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:20.993038 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:20.993468 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:20.993473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:20.993476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:20.993730 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:20.993865 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:20.994518 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 19:26:20.994731 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:26:20.995998 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:20.978331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37687 (local address 127.19.68.3:41445)
0810 19:26:20.979120 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:20.979162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:20.980770 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:20.981482 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:20.981490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:20.981995 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:20.982529 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:20.982545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.983598 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.983602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:20.985030 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:20.985046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:20.985872 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:20.985880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:20.986142 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:20.986857 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:20.986885 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:20.989207 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:20.992373 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:20.992382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:20.992397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:20.992790 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:20.993166 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:20.993171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:20.993175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:20.993363 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:20.993865 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:20.993872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:20.994173 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 19:26:20.994408 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 19:26:20.994889 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":349,"threads_started":1}
I20260810 19:26:20.997217 30262 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37687
I20260810 19:26:20.997493 30262 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:20.998171 30262 heartbeater.cc:511] Master 127.19.68.62:37687 requested a full tablet report, sending...
I20260810 19:26:20.999608 29438 ts_manager.cc:194] Registered new tserver with Master: cb5c8b1b0c8a4154b212babc942eee6e (127.19.68.3:42903)
I20260810 19:26:21.000537 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:26:21.000532 29438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:41445
I20260810 19:26:21.009456 29776 heartbeater.cc:503] Master 127.19.68.62:37687 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/test-admin.keytab.
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.031715 19728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 19:26:21.036769 30278 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:21.034356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.19.68.1:38583)
0810 19:26:21.035012 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.035019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.035047 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:26:21.035120 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.035126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.035341 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.035479 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.036370 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45748: BlockingRecv error: recv got EOF from 127.0.0.1:45748 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":337,"threads_started":1}
W20260810 19:26:21.036775 19728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc' of type FILE
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.038286 19728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.039556 19728 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:21.054674 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.041239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.19.68.1:38583)
0810 19:26:21.041419 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.041424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.041523 (+ 99us) server_negotiation.cc:407] Connection header received
0810 19:26:21.041644 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.041647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.041702 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.041797 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.042775 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.043803 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.044984 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.045629 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.045735 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.049155 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.049167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.049180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.049212 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.051865 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.052567 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.052575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.052577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.052704 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.053081 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.053109 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.053113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.053496 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.053722 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.054054 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.054371 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 19:26:21.060139 30278 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:21.058467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.19.68.1:38583)
0810 19:26:21.058670 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.058676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.058910 (+ 234us) server_negotiation.cc:407] Connection header received
0810 19:26:21.059052 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.059056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.059118 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.059195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.059932 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45774: BlockingRecv error: recv got EOF from 127.0.0.1:45774 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 19:26:21.060330 19728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc' of type FILE
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.061823 19728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.063023 19728 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:21.075827 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.064183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.19.68.1:38583)
0810 19:26:21.064386 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.064392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.064610 (+ 218us) server_negotiation.cc:407] Connection header received
0810 19:26:21.064717 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.064722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.064779 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.064859 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.065836 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.066559 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.067687 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.068100 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.068185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.071403 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.071407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.071410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.071441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.073721 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.074252 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.074255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.074258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.074391 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.074648 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.074650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.074652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.074916 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.075111 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.075392 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.075546 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 19:26:21.079780 30278 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:21.078274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.19.68.1:38583)
0810 19:26:21.078468 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.078474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.078655 (+ 181us) server_negotiation.cc:407] Connection header received
0810 19:26:21.078792 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.078797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.078869 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.078948 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.079582 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45792: BlockingRecv error: recv got EOF from 127.0.0.1:45792 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 19:26:21.079933 19728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389929706493-19728-0/krb5kdc/krb5cc' of type FILE
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.081364 19728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:21.082525 19728 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:21.095608 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.083659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.19.68.1:38583)
0810 19:26:21.083869 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.083874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.084139 (+ 265us) server_negotiation.cc:407] Connection header received
0810 19:26:21.084281 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.084284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.084334 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.084406 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.085342 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.086052 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.087212 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.087597 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.087699 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.090869 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.090873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.090876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.090922 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.093264 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.093836 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.093849 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.093852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.093997 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.094338 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.094342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.094345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.094586 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.094921 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.095114 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.095337 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:26:21.097234 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 29538
I20260810 19:26:21.107832 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.097892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.19.68.1:38583)
0810 19:26:21.098030 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.098034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.098397 (+ 363us) server_negotiation.cc:407] Connection header received
0810 19:26:21.098512 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.098515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.098563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.098632 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.099447 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.100098 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.101210 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.101614 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.101705 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.103433 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.103437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.103440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.103466 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.105772 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.106292 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.106295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.106297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.106414 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.106729 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.106733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.106735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.107017 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.107222 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.107414 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.107566 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:26:21.108353 29769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:21.189785 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 29538
I20260810 19:26:21.214538 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 29789
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:26:21.230078 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.215481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.19.68.2:41635)
0810 19:26:21.216034 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.216041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.216066 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:26:21.216133 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.216140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.216353 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.216525 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.217670 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.218732 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.219962 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.220527 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.220645 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.224429 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.224441 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.224455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.224495 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.227322 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.227983 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.227990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.227993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.228118 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.228514 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.228518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.228520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.228870 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.229195 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.229515 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.229777 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":271,"threads_started":1}
I20260810 19:26:21.231447 30014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:21.312053 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 29789
I20260810 19:26:21.332844 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 30034
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:26:21.348559 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:21.333902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.19.68.3:42903)
0810 19:26:21.334593 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:21.334600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:21.334630 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:26:21.334692 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:21.334698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:21.334900 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:21.335033 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:21.336126 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.337253 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.338500 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:21.339040 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:21.339137 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:21.343095 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:21.343107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:21.343121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:21.343153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:21.345889 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.346569 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.346576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.346579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.346703 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:21.347059 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:21.347062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:21.347064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:21.347391 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:21.347684 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:21.347975 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 19:26:21.348225 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":376,"threads_started":1}
I20260810 19:26:21.349977 30253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:21.436045 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 30034
I20260810 19:26:21.460739 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 29391
I20260810 19:26:21.462064 29489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:21.564657 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 29391
2026-08-10T19:26:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4133 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[30316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[30316](info): set up 2 sockets
Aug 10 19:26:21 dist-test-slave-2j7r krb5kdc[30316](info): commencing operation
krb5kdc: starting...
W20260810 19:26:22.839658 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:23 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab.
Entry for principal oryx/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab.
I20260810 19:26:23.088384 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:42309
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab
--principal=oryx/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:23.423761 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:23.424043 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:23.424180 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:23.431643 30335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:23.431751 30335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:23.431831 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:23.431913 30335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:23.431975 30335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:23.441704 30335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab
--principal=oryx/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:23.442974 30335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:23.444217 30335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:23.455504 30343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:23.455503 30344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:23.455536 30346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:23.456357 30335 server_base.cc:1034] running on GCE node
I20260810 19:26:23.457054 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:23.458364 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:23.459573 30335 hybrid_clock.cc:648] HybridClock initialized: now 1786389983459527 us; error 33 us; skew 500 ppm
I20260810 19:26:23.459898 30335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:23 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:23.462908 30335 init.cc:377] Logged in from keytab as oryx/127.19.68.62@KRBTEST.COM (short username oryx)
I20260810 19:26:23.464303 30335 webserver.cc:460] Webserver started at http://127.19.68.62:40405/ using document root <none> and password file <none>
I20260810 19:26:23.464951 30335 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:23.465071 30335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:23.465402 30335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:23.468339 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "57286387165c4738b948e530dd4a0bef"
format_stamp: "Formatted at 2026-08-10 19:26:23 on dist-test-slave-2j7r"
server_key: "71743dd986a060645abca4467f714db6"
server_key_iv: "73263f15638008c5f02fc733ee559e52"
server_key_version: "encryptionkey@0"
I20260810 19:26:23.469177 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "57286387165c4738b948e530dd4a0bef"
format_stamp: "Formatted at 2026-08-10 19:26:23 on dist-test-slave-2j7r"
server_key: "71743dd986a060645abca4467f714db6"
server_key_iv: "73263f15638008c5f02fc733ee559e52"
server_key_version: "encryptionkey@0"
I20260810 19:26:23.474251 30335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 19:26:23.477620 30361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:23.478932 30335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:26:23.479095 30335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "57286387165c4738b948e530dd4a0bef"
format_stamp: "Formatted at 2026-08-10 19:26:23 on dist-test-slave-2j7r"
server_key: "71743dd986a060645abca4467f714db6"
server_key_iv: "73263f15638008c5f02fc733ee559e52"
server_key_version: "encryptionkey@0"
I20260810 19:26:23.479266 30335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:23.510946 30335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:23.514853 30335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:23.515121 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:23.532096 30335 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:42309
I20260810 19:26:23.532130 30447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:42309 every 8 connection(s)
I20260810 19:26:23.533461 30335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:23.537192 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:23.542018 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 30335
I20260810 19:26:23.542234 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:23.543074 19728 external_mini_cluster.cc:1428] Setting key 5b5e17f3ac8a4a4e70968e6c555b679c
I20260810 19:26:23.548568 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Bootstrap starting.
I20260810 19:26:23.552129 30449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:23.553154 30449 log.cc:824] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:23.555985 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: No bootstrap required, opened a new log
Aug 10 19:26:23 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:23.564026 30449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } }
I20260810 19:26:23.564419 30449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:23.564544 30449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57286387165c4738b948e530dd4a0bef, State: Initialized, Role: FOLLOWER
I20260810 19:26:23.566105 30449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } }
I20260810 19:26:23.566416 30449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:23.566596 30449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:23.566804 30449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:23.567363 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:23.545384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.19.68.62:42309)
0810 19:26:23.546329 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:23.546346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:23.546399 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:26:23.548199 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:23.548233 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:23.548627 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:23.549010 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:23.550866 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:23.552157 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:23.553569 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:23.554294 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:23.554461 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:23.558402 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:23.558413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:23.558428 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:23.558465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:23.561937 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:23.562734 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:23.562741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:23.562752 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:23.562937 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:23.563393 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:23.563397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:23.563399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:23.563987 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:23.564302 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:23.565449 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0810 19:26:23.565808 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":468,"threads_started":1}
I20260810 19:26:23.569537 30449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } }
I20260810 19:26:23.570010 30449 leader_election.cc:302] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57286387165c4738b948e530dd4a0bef; no voters:
I20260810 19:26:23.570659 30449 leader_election.cc:288] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:23.570770 30466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:23.571414 30466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 LEADER]: Becoming Leader. State: Replica: 57286387165c4738b948e530dd4a0bef, State: Running, Role: LEADER
I20260810 19:26:23.571965 30466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } }
I20260810 19:26:23.572858 30449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:23.574934 30469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57286387165c4738b948e530dd4a0bef. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } }
I20260810 19:26:23.574972 30468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } }
I20260810 19:26:23.575393 30469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:23.575395 30468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:23.580209 30478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:23.586505 30478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:23.595533 30478 catalog_manager.cc:1347] Generated new cluster ID: a0a171afd5874fdaafd1137337b2a4a2
I20260810 19:26:23.595642 30478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:23.608392 30478 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:23.610157 30478 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:23.619652 30478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Generated new TSK 0
I20260810 19:26:23.620416 30478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
I20260810 19:26:23.688855 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:24.033018 30499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:24.033298 30499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:24.033416 30499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:24.040488 30499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:24.040591 30499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:24.040730 30499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:24.050287 30499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:24.051661 30499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:24.053015 30499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:24.066076 30510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:24.066037 30513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:24.066167 30511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:24.067293 30499 server_base.cc:1034] running on GCE node
I20260810 19:26:24.067909 30499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:24.068821 30499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:24.070021 30499 hybrid_clock.cc:648] HybridClock initialized: now 1786389984069956 us; error 63 us; skew 500 ppm
I20260810 19:26:24.070322 30499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:24 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389984, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:24.073436 30499 init.cc:377] Logged in from keytab as oryx/127.19.68.1@KRBTEST.COM (short username oryx)
I20260810 19:26:24.074971 30499 webserver.cc:460] Webserver started at http://127.19.68.1:45241/ using document root <none> and password file <none>
I20260810 19:26:24.075623 30499 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:24.075726 30499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:24.076016 30499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:24.078984 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "0595e8ff3ebc42a39d0323a61440eb12"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "7eb156ada7e1cc2628fc8264bcb36374"
server_key_iv: "e0c658da202752ff797af0e68e2d3baf"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.079917 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "0595e8ff3ebc42a39d0323a61440eb12"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "7eb156ada7e1cc2628fc8264bcb36374"
server_key_iv: "e0c658da202752ff797af0e68e2d3baf"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.085170 30499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:26:24.088443 30522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:24.089720 30499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:24.089854 30499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "0595e8ff3ebc42a39d0323a61440eb12"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "7eb156ada7e1cc2628fc8264bcb36374"
server_key_iv: "e0c658da202752ff797af0e68e2d3baf"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.090034 30499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:24.105286 30499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:24.109026 30499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:24.109304 30499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:24.110204 30499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:24.111498 30499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:24.111572 30499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:24.111652 30499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:24.111697 30499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:24.142416 30499 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34767
I20260810 19:26:24.142454 30724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34767 every 8 connection(s)
I20260810 19:26:24.143664 30499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:24.148487 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 30499
I20260810 19:26:24.148699 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:24.149564 19728 external_mini_cluster.cc:1428] Setting key 549b7c878dcbe60c02d6a84e9699495e
Aug 10 19:26:24 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389984, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:24.163419 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:24.146271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:52597 (local address 127.19.68.62:42309)
0810 19:26:24.146731 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:24.146735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:24.147486 (+ 751us) server_negotiation.cc:407] Connection header received
0810 19:26:24.149893 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:24.149898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:24.149965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:24.150057 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:24.151542 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.152279 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.153867 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.154272 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.154391 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:24.157871 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:24.157876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:24.157880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:24.157912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:24.160411 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:24.161123 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:24.161130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:24.161133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:24.161257 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:24.161685 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:24.161690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:24.161693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:24.161975 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:24.162110 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:24.162817 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 19:26:24.163048 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260810 19:26:24.164316 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:24.146626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:52597)
0810 19:26:24.147324 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:24.147371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:24.149580 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:24.150311 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:24.150319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:24.150803 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:24.151362 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:24.151380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.152440 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.152444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:24.153702 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:24.153716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.154494 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.154502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:24.154731 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:24.155336 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:24.155358 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:24.157653 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:24.160545 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:24.160554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:24.160569 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:24.160979 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:24.161363 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:24.161368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:24.161372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:24.161566 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:24.162101 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:24.162109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:24.162436 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 19:26:24.162659 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 19:26:24.163168 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260810 19:26:24.165853 30725 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:24.166165 30725 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:24.167008 30725 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:24.169445 30391 ts_manager.cc:194] Registered new tserver with Master: 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767)
I20260810 19:26:24.171324 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:52597
WARNING: no policy specified for oryx/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
I20260810 19:26:24.206280 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:24.546875 30735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:24.547224 30735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:24.547390 30735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:24.557543 30735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:24.557658 30735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:24.557793 30735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:24.567473 30735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:24.568905 30735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:24.570616 30735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:24.584261 30749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:24.584223 30745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:24.584348 30744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:24.585053 30735 server_base.cc:1034] running on GCE node
I20260810 19:26:24.585747 30735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:24.586745 30735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:24.588006 30735 hybrid_clock.cc:648] HybridClock initialized: now 1786389984587940 us; error 36 us; skew 500 ppm
I20260810 19:26:24.588390 30735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:24 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389984, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:24.595768 30735 init.cc:377] Logged in from keytab as oryx/127.19.68.2@KRBTEST.COM (short username oryx)
I20260810 19:26:24.597306 30735 webserver.cc:460] Webserver started at http://127.19.68.2:43641/ using document root <none> and password file <none>
I20260810 19:26:24.598016 30735 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:24.598125 30735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:24.598373 30735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:24.601347 30735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "de73a592a4e94614a914a184aceb2a87"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "8e12d866bfb62f28b39757a06f1b32d0"
server_key_iv: "d1e2c32950e54774590972093ae32290"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.602097 30735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "de73a592a4e94614a914a184aceb2a87"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "8e12d866bfb62f28b39757a06f1b32d0"
server_key_iv: "d1e2c32950e54774590972093ae32290"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.607148 30735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:24.610435 30758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:24.611682 30735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:26:24.611832 30735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "de73a592a4e94614a914a184aceb2a87"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "8e12d866bfb62f28b39757a06f1b32d0"
server_key_iv: "d1e2c32950e54774590972093ae32290"
server_key_version: "encryptionkey@0"
I20260810 19:26:24.612023 30735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:24.638682 30735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:24.642480 30735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:24.642719 30735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:24.643618 30735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:24.644892 30735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:24.644969 30735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:24.645061 30735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:24.645129 30735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 19:26:24.675156 30735 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:33205
I20260810 19:26:24.675197 30945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:33205 every 8 connection(s)
I20260810 19:26:24.676388 30735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:24.681124 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 30735
I20260810 19:26:24.681303 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:24.681901 19728 external_mini_cluster.cc:1428] Setting key a438f24c959c050299bd7d8a453118fa
Aug 10 19:26:24 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389984, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:24.696817 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:24.678806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49969 (local address 127.19.68.62:42309)
0810 19:26:24.679210 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:24.679215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:24.680045 (+ 830us) server_negotiation.cc:407] Connection header received
0810 19:26:24.682414 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:24.682418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:24.682492 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:24.682622 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:24.684186 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.684872 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.687011 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.687428 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.687524 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:24.691280 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:24.691285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:24.691288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:24.691320 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:24.693824 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:24.694586 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:24.694592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:24.694595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:24.694759 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:24.695138 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:24.695142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:24.695145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:24.695486 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:24.695628 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:24.696229 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 19:26:24.696458 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":203,"threads_started":1}
I20260810 19:26:24.697533 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:24.679193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:49969)
0810 19:26:24.679874 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:24.679919 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:24.682094 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:24.682830 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:24.682842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:24.683380 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:24.684020 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:24.684043 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.685025 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.685030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:24.686858 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:24.686878 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:24.687635 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:24.687641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:24.687892 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:24.688527 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:24.688555 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:24.691065 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:24.693937 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:24.693944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:24.693958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:24.694472 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:24.694857 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:24.694860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:24.694862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:24.695037 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:24.695589 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:24.695594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:24.695881 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 19:26:24.696091 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 19:26:24.696458 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260810 19:26:24.698843 30946 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:24.699167 30946 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:24.700063 30946 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:24.701517 30391 ts_manager.cc:194] Registered new tserver with Master: de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:24.702306 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:49969
WARNING: no policy specified for oryx/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
I20260810 19:26:24.737886 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:25.058736 30958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:25.058992 30958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:25.059116 30958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:25.066274 30958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:25.066403 30958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:25.066560 30958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:25.076305 30958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:25.077621 30958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:25.078891 30958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:25.091485 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:25.091517 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:25.091734 30963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:25.092444 30958 server_base.cc:1034] running on GCE node
I20260810 19:26:25.092974 30958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:25.093918 30958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:25.095115 30958 hybrid_clock.cc:648] HybridClock initialized: now 1786389985095074 us; error 37 us; skew 500 ppm
I20260810 19:26:25.095408 30958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389985, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:25.098410 30958 init.cc:377] Logged in from keytab as oryx/127.19.68.3@KRBTEST.COM (short username oryx)
I20260810 19:26:25.099745 30958 webserver.cc:460] Webserver started at http://127.19.68.3:40701/ using document root <none> and password file <none>
I20260810 19:26:25.100345 30958 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:25.100454 30958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:25.100697 30958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:25.103535 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "ec40d07ba1ab434988bd05c03777fcce"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "0c4c16c8ce22cd504db951c85f220512"
server_key_iv: "00cdde44220a924c271697a95bf57978"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.104225 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "ec40d07ba1ab434988bd05c03777fcce"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "0c4c16c8ce22cd504db951c85f220512"
server_key_iv: "00cdde44220a924c271697a95bf57978"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.109012 30958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 19:26:25.112411 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:25.113783 30958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:26:25.113920 30958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "ec40d07ba1ab434988bd05c03777fcce"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "0c4c16c8ce22cd504db951c85f220512"
server_key_iv: "00cdde44220a924c271697a95bf57978"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.114092 30958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:25.143085 30958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:25.147343 30958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:25.147621 30958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:25.148541 30958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:25.149880 30958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:25.149955 30958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:25.150065 30958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:25.150125 30958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:25.174938 30725 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
I20260810 19:26:25.181200 30958 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42349
I20260810 19:26:25.181310 31189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42349 every 8 connection(s)
I20260810 19:26:25.182479 30958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:25.189245 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 30958
I20260810 19:26:25.189442 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:25.189898 19728 external_mini_cluster.cc:1428] Setting key 26663ce2e408e77a67937be275082f38
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389985, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:25.201272 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.184919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:53101 (local address 127.19.68.62:42309)
0810 19:26:25.185115 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.185121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.186265 (+ 1144us) server_negotiation.cc:407] Connection header received
0810 19:26:25.188041 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.188047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.188131 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.188218 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.189780 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.190599 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.192095 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.192503 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.192610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.195913 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.195918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.195921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.195953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.198352 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.199049 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.199053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.199055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.199179 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.199593 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.199596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.199598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.199871 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.200008 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.200697 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.200934 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:26:25.202211 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.185410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:53101)
0810 19:26:25.186115 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.186152 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.187780 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.188463 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.188470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.188952 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.189618 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.189639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.190752 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.190757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.191957 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.191970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.192714 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.192720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.192964 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.193549 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.193571 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.195723 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.198499 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.198507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.198522 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.198926 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.199281 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.199286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.199289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.199479 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.199996 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.200004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.200319 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.200546 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.201072 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
I20260810 19:26:25.203699 31190 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:25.204026 31190 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:25.204773 31190 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:25.206189 30391 ts_manager.cc:194] Registered new tserver with Master: ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349)
I20260810 19:26:25.206954 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:53101
I20260810 19:26:25.217221 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.61.keytab.
Entry for principal oryx/127.19.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.61.keytab.
I20260810 19:26:25.271869 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:44511
--webserver_interface=127.19.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.61.keytab
--principal=oryx/127.19.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:42309,127.19.68.61:44511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:25.600800 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:25.601105 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:25.601241 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:25.608422 31198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:25.608537 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:25.608615 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:25.608693 31198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:25.608754 31198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:25.618793 31198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:42309,127.19.68.61:44511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.61.keytab
--principal=oryx/127.19.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
W20260810 19:26:25.619144 31198 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.68.62:42309,127.19.68.61:44511'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 19:26:25.620162 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:25.621708 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:25.633045 31209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:25.633190 31207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:25.633390 31204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:25.634311 31198 server_base.cc:1034] running on GCE node
I20260810 19:26:25.635026 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:25.636379 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:25.637602 31198 hybrid_clock.cc:648] HybridClock initialized: now 1786389985637531 us; error 39 us; skew 500 ppm
I20260810 19:26:25.637913 31198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389985, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:25.640969 31198 init.cc:377] Logged in from keytab as oryx/127.19.68.61@KRBTEST.COM (short username oryx)
I20260810 19:26:25.642467 31198 webserver.cc:460] Webserver started at http://127.19.68.61:34659/ using document root <none> and password file <none>
I20260810 19:26:25.643131 31198 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:25.643217 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:25.643483 31198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:25.646595 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/instance:
uuid: "8ac017532d5f4f189775bbefda2e28ba"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "1d570f6fb043f7ca194756a572294ef4"
server_key_iv: "b42a9103629726ebb3329684cdece9bb"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.647361 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal/instance:
uuid: "8ac017532d5f4f189775bbefda2e28ba"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "1d570f6fb043f7ca194756a572294ef4"
server_key_iv: "b42a9103629726ebb3329684cdece9bb"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.652576 31198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:26:25.656001 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:25.657353 31198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:25.657486 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "8ac017532d5f4f189775bbefda2e28ba"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "1d570f6fb043f7ca194756a572294ef4"
server_key_iv: "b42a9103629726ebb3329684cdece9bb"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.657656 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:25.670181 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:25.674204 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:25.674456 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:25.692984 31198 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.61:44511
I20260810 19:26:25.693003 31303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.61:44511 every 8 connection(s)
I20260810 19:26:25.694478 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
I20260810 19:26:25.695117 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 31198
I20260810 19:26:25.698804 31305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:25.705181 30946 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
I20260810 19:26:25.707022 31305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389985, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.61@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:25.719439 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.697708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.19.68.61:44511)
0810 19:26:25.698411 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.698422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.698464 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:26:25.700051 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.700071 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.700418 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.700767 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.702722 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.704111 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.705923 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.706904 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.707071 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.711757 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.711768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.711781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.711831 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.715035 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.715693 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.715702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.715709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.715933 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.716348 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.716353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.716356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.717023 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.717372 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.717823 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.718228 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260810 19:26:25.724229 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.709418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.19.68.62:42309)
0810 19:26:25.709885 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.709891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.710396 (+ 505us) server_negotiation.cc:407] Connection header received
0810 19:26:25.710555 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.710559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.710627 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.710727 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.712104 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.713061 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.714516 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.714941 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.715046 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.718939 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.718944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.718948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.718980 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.721540 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.722298 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.722302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.722304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.722419 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.722776 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.722780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.722782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.723077 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.723224 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.723647 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.723861 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
I20260810 19:26:25.724267 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.709439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:42248)
0810 19:26:25.710221 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.710294 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.710451 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.710822 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.710828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.711632 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.711935 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.711944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.713234 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.713239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.714377 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.714390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.715117 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.715124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.715288 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.715871 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.715893 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.718753 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.721663 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.721674 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.721681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.722180 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.722531 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.722536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.722540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.722672 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.723219 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.723226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.723372 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.723534 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.723851 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"spinlock_wait_cycles":14592,"thread_start_us":414,"threads_started":1}
I20260810 19:26:25.725910 31305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389985, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.61@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
I20260810 19:26:25.737143 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.726485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51680)
0810 19:26:25.726633 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.726648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.726726 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.727014 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.727017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.727314 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.727441 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.727449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.728331 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.728335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.729275 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.729290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.730021 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.730025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.730199 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.730681 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.730697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.732664 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.735151 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.735155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.735158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.735532 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.735855 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.735858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.735860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.735991 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.736417 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.736420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.736549 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.736658 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.736864 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:26:25.737373 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.726694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.19.68.61:44511)
0810 19:26:25.726817 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.726821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.726835 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:26:25.726869 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.726872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.726922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.726991 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.727556 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.728207 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.729438 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.729890 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.730007 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.732853 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.732857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.732860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.732887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.735034 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.735625 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.735628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.735631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.735758 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.736076 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.736079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.736080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.736316 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.736424 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.736736 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.737008 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":41}
I20260810 19:26:25.742172 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Bootstrap starting.
I20260810 19:26:25.745235 31305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:25.746263 31305 log.cc:824] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:25.748915 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: No bootstrap required, opened a new log
I20260810 19:26:25.755718 31305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } }
I20260810 19:26:25.756033 31305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:25.756121 31305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac017532d5f4f189775bbefda2e28ba, State: Initialized, Role: FOLLOWER
I20260810 19:26:25.756645 31305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } }
I20260810 19:26:25.757485 31321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } }
I20260810 19:26:25.757761 31321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:25.758415 31305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:26:25.766723 31334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:26:25.766855 31334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:26:25.769966 31198 master_runner.cc:428] Detected that this master 8ac017532d5f4f189775bbefda2e28ba is joining an existing cluster
I20260810 19:26:25.770063 31198 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.68.61:44511
I20260810 19:26:25.792209 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.774112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.19.68.62:42309)
0810 19:26:25.774273 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.774278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.774983 (+ 705us) server_negotiation.cc:407] Connection header received
0810 19:26:25.775243 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.775248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.775344 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.775418 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.776979 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.778113 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.780115 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.780765 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.780867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.784326 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.784333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.784338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.784377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.788745 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.789697 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.789703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.789707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.789880 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.790395 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.790401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.790404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.790800 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.790931 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.791628 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.791879 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:26:25.793232 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.774202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:42252)
0810 19:26:25.774883 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.774902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.775092 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.775645 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.775650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.776392 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.776800 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.776814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.778297 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.778303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.779943 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.779959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.781198 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.781205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.781378 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.781975 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.781998 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.784120 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.788910 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.788914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.788922 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.789400 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.790073 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.790077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.790081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.790255 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.791037 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.791041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.791251 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.792463 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.792784 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for oryx/127.19.68.60@KRBTEST.COM; defaulting to no policy
I20260810 19:26:25.816170 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.775066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.19.68.61:44511)
0810 19:26:25.780860 (+ 5794us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.780867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.793636 (+ 12769us) server_negotiation.cc:407] Connection header received
0810 19:26:25.793794 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.793801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.793887 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.794009 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.795042 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.796120 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.801345 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.802013 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.802105 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.807123 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.807129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.807132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.807163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.812232 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.812995 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.813000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.813004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.813188 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.813599 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.813603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.813606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.813963 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.814133 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.814972 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.815697 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":457,"server-negotiator.queue_time_us":5637}
I20260810 19:26:25.816871 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.775181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51692)
0810 19:26:25.793545 (+ 18364us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.793564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.793675 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.794072 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.794077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.794642 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.794880 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.794893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.796373 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.796379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.801202 (+ 4823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.801218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.802339 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.802344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.802508 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.805632 (+ 3124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.805653 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.806956 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.812377 (+ 5421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.812382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.812386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.812865 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.813306 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.813309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.813313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.813485 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.814186 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.814191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.814355 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.814534 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.815127 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18260,"mutex_wait_us":1,"thread_start_us":21595,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.60@KRBTEST.COM" created.
Aug 10 19:26:25 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.62@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
I20260810 19:26:25.838840 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.809695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51702)
0810 19:26:25.817553 (+ 7858us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.817598 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.817854 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.818786 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.818794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.819739 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.820099 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.820113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.823561 (+ 3448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.823567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.825243 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.825261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.826437 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.826444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.826677 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.828421 (+ 1744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.828446 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.831455 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:25.835570 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.835584 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.835591 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.836195 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.836653 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:25.836658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:25.836662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:25.836848 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:25.837636 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:25.837644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:25.837856 (+ 212us) client_negotiation.cc:769] Sending connection context
0810 19:26:25.838060 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 19:26:25.838406 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7439,"thread_start_us":796,"threads_started":1}
I20260810 19:26:25.839931 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.810086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.19.68.61:44511)
0810 19:26:25.816300 (+ 6214us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.816307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.818115 (+ 1808us) server_negotiation.cc:407] Connection header received
0810 19:26:25.818249 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.818253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.818336 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.818432 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.820306 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.823393 (+ 3087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.825419 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.826081 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.826200 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.831711 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.831718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.831724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.831773 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:25.835404 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.836342 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.836347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.836350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.836512 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:25.836983 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:25.836988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:25.836992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:25.837404 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:25.837543 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:25.839287 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0810 19:26:25.839512 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6073,"thread_start_us":5909,"threads_started":1}
I20260810 19:26:25.846560 30391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.68.61:44511: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba"
member_type: NON_VOTER
last_known_addr {
host: "127.19.68.61"
port: 44511
}
attrs {
promote: true
}
}
dest_uuid: "57286387165c4738b948e530dd4a0bef"
cas_config_opid_index: -1
I20260810 19:26:25.847955 30391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.60.keytab.
Entry for principal oryx/127.19.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.60.keytab.
I20260810 19:26:25.865124 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:44749
--webserver_interface=127.19.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.60.keytab
--principal=oryx/127.19.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:26:25.866510 31359 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.867722 31267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:25.871889 31267 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 19:26:25.873174 31358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:25.876538 31359 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.68.61:44511
I20260810 19:26:25.877522 31198 master.cc:537] Master@127.19.68.61:44511 shutting down...
I20260810 19:26:25.882228 31360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.886161 31360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:25.909646 31267 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.915654 31321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57286387165c4738b948e530dd4a0bef. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.915346 31369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.915959 31321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:25.915997 31369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:25.914631 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:25.916340 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:25.943878 31198 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Raft consensus shutting down.
I20260810 19:26:25.944532 31198 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Raft consensus is shut down!
I20260810 19:26:25.944774 31198 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: stopping tablet replica
I20260810 19:26:25.954730 31198 master.cc:552] Master@127.19.68.61:44511 shutdown complete.
I20260810 19:26:25.954926 31198 master_runner.cc:305] Clearing existing system tablet
I20260810 19:26:25.956773 31198 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:25.956916 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:25.960301 31198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:26:25.962225 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:25.962613 31198 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:25.963400 31198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:26:25.963551 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "8ac017532d5f4f189775bbefda2e28ba"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "1d570f6fb043f7ca194756a572294ef4"
server_key_iv: "b42a9103629726ebb3329684cdece9bb"
server_key_version: "encryptionkey@0"
I20260810 19:26:25.968326 31198 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:25.975541 31198 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 19:26:25.975914 31198 log.cc:1196] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 19:26:25.976517 31198 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Deleting consensus metadata
I20260810 19:26:25.976835 31198 master_runner.cc:315] Copying system tablet from 127.19.68.62:42309
I20260810 19:26:25.979590 31198 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: tablet copy: Beginning tablet copy session from remote peer at address 127.19.68.62:42309
I20260810 19:26:26.003844 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.980560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.19.68.62:42309)
0810 19:26:25.981207 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:25.981214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:25.989736 (+ 8522us) server_negotiation.cc:407] Connection header received
0810 19:26:25.989945 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:25.989952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:25.990053 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:25.990173 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:25.991199 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.992312 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.994132 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.994799 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.994930 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.997281 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:25.997288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:25.997293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:25.997335 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.000743 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.001603 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.001610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.001614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.001799 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.002241 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.002246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.002249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.002616 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.002790 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.003237 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.003469 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260810 19:26:26.003877 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:25.980467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:42260)
0810 19:26:25.989583 (+ 9116us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:25.989604 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:25.989789 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:25.990217 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:25.990222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:25.990754 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:25.991018 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.991032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.992454 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.992460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.993972 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:25.993988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:25.994978 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:25.994987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:25.995162 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:25.995865 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:25.995890 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:25.997028 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.000891 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.000897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.000901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.001458 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.001919 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.001924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.001927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.002115 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.002751 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.002756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.002945 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.003101 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.003427 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8948,"thread_start_us":8788,"threads_started":1}
I20260810 19:26:26.005163 30418 tablet_copy_service.cc:140] P 57286387165c4738b948e530dd4a0bef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8ac017532d5f4f189775bbefda2e28ba ({username='oryx', principal='oryx/127.19.68.61@KRBTEST.COM'} at 127.0.0.1:42260)
I20260810 19:26:26.005439 30418 tablet_copy_service.cc:161] P 57286387165c4738b948e530dd4a0bef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8ac017532d5f4f189775bbefda2e28ba at {username='oryx', principal='oryx/127.19.68.61@KRBTEST.COM'} at 127.0.0.1:42260: session id = 8ac017532d5f4f189775bbefda2e28ba-00000000000000000000000000000000
I20260810 19:26:26.010138 30418 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 19:26:26.012389 31198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 19:26:26.019301 31198 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: tablet copy: Starting download of 0 data blocks...
I20260810 19:26:26.019629 31198 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: tablet copy: Starting download of 1 WAL segments...
I20260810 19:26:26.022589 31198 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 19:26:26.027001 30418 tablet_copy_service.cc:342] P 57286387165c4738b948e530dd4a0bef: Request end of tablet copy session 8ac017532d5f4f189775bbefda2e28ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.68.61@KRBTEST.COM'} at 127.0.0.1:42260
I20260810 19:26:26.027204 30418 tablet_copy_service.cc:433] P 57286387165c4738b948e530dd4a0bef: ending tablet copy session 8ac017532d5f4f189775bbefda2e28ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8ac017532d5f4f189775bbefda2e28ba
I20260810 19:26:26.039952 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:26.042698 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:26.046490 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:26.053263 31390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:26.058454 31198 server_base.cc:1034] running on GCE node
I20260810 19:26:26.058925 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:26.059599 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:26.060837 31198 hybrid_clock.cc:648] HybridClock initialized: now 1786389986060759 us; error 48 us; skew 500 ppm
I20260810 19:26:26.061077 31198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:26.063355 31198 init.cc:377] Logged in from keytab as oryx/127.19.68.61@KRBTEST.COM (short username oryx)
I20260810 19:26:26.064682 31198 webserver.cc:460] Webserver started at http://127.19.68.61:44671/ using document root <none> and password file <none>
I20260810 19:26:26.065101 31198 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:26.065192 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:26.068605 31198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:26:26.070832 31401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:26.071341 31198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 19:26:26.071520 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "8ac017532d5f4f189775bbefda2e28ba"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "1d570f6fb043f7ca194756a572294ef4"
server_key_iv: "b42a9103629726ebb3329684cdece9bb"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.071717 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:26.097488 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:26.101874 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:26.139600 31198 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.61:44511
I20260810 19:26:26.139767 31484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.61:44511 every 8 connection(s)
I20260810 19:26:26.140633 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
I20260810 19:26:26.149900 31486 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:26:26.153684 31486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Bootstrap starting.
I20260810 19:26:26.167886 31486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:26:26.168440 31486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: Bootstrap complete.
I20260810 19:26:26.169626 31486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } }
I20260810 19:26:26.169917 31486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ac017532d5f4f189775bbefda2e28ba, State: Initialized, Role: LEARNER
I20260810 19:26:26.170245 31486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } }
I20260810 19:26:26.171423 31486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:26.171147 31491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.172703 31491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:26.179703 31505 catalog_manager.cc:1259] Loaded cluster ID: a0a171afd5874fdaafd1137337b2a4a2
I20260810 19:26:26.179860 31505 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: loading cluster ID for follower catalog manager: success
I20260810 19:26:26.183362 31505 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: acquiring CA information for follower catalog manager: success
I20260810 19:26:26.186223 31505 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:26:26.209429 31190 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
I20260810 19:26:26.274688 31359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.275024 31359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
W20260810 19:26:26.278859 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:26.279206 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:26.279361 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 19:26:26.281991 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.275086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51714)
0810 19:26:26.275309 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.275327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.275433 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.276368 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.276374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.276410 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:26.276598 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.276609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.277970 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.277976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.279961 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.279979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.281188 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.281194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.281392 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.281459 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.281565 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.281725 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 19:26:26.282019 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.275433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.19.68.61:44511)
0810 19:26:26.276028 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.276033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.276053 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:26:26.276112 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.276115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.276181 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.276276 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:26.276746 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.277829 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.280099 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.281006 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.281136 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.281320 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.281580 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.281689 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
I20260810 19:26:26.282932 31491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57286387165c4738b948e530dd4a0bef. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.283205 31491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:26.283870 31359 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: attempting to promote NON_VOTER 8ac017532d5f4f189775bbefda2e28ba to VOTER
I20260810 19:26:26.284814 31359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } }
I20260810 19:26:26.286506 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 LEARNER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 19:26:26.287037 31360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260810 19:26:26.288620 31363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:26.288722 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:26.288794 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:26.288872 31363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:26.288929 31363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:26.289794 31359 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } }
I20260810 19:26:26.290343 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } }
I20260810 19:26:26.293011 31491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } }
I20260810 19:26:26.293238 31491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:26.293876 31358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } }
I20260810 19:26:26.294116 31358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:26.294466 31358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } }
I20260810 19:26:26.294628 31358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:26.300292 31363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.60.keytab
--principal=oryx/127.19.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:26.301646 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:26.302906 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:26.313961 31518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:26.314064 31520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:26.314038 31522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:26.314884 31363 server_base.cc:1034] running on GCE node
I20260810 19:26:26.315666 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:26.317126 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:26.318346 31363 hybrid_clock.cc:648] HybridClock initialized: now 1786389986318301 us; error 43 us; skew 500 ppm
I20260810 19:26:26.318681 31363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:26.321616 31363 init.cc:377] Logged in from keytab as oryx/127.19.68.60@KRBTEST.COM (short username oryx)
I20260810 19:26:26.322985 31363 webserver.cc:460] Webserver started at http://127.19.68.60:42241/ using document root <none> and password file <none>
I20260810 19:26:26.323587 31363 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:26.323684 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:26.323925 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:26.326759 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/instance:
uuid: "f39095eeb1164695aad63abd8ff8405d"
format_stamp: "Formatted at 2026-08-10 19:26:26 on dist-test-slave-2j7r"
server_key: "228550bf7c9fe0d782bcbef22465b769"
server_key_iv: "71aca68384281d84da1827ed2a6583a2"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.327471 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal/instance:
uuid: "f39095eeb1164695aad63abd8ff8405d"
format_stamp: "Formatted at 2026-08-10 19:26:26 on dist-test-slave-2j7r"
server_key: "228550bf7c9fe0d782bcbef22465b769"
server_key_iv: "71aca68384281d84da1827ed2a6583a2"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.332338 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 19:26:26.335664 31534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:26.336938 31363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:26.337078 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "f39095eeb1164695aad63abd8ff8405d"
format_stamp: "Formatted at 2026-08-10 19:26:26 on dist-test-slave-2j7r"
server_key: "228550bf7c9fe0d782bcbef22465b769"
server_key_iv: "71aca68384281d84da1827ed2a6583a2"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.337301 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:26.354523 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:26.358179 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:26.358412 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:26.376169 31363 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.60:44749
I20260810 19:26:26.376207 31614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.60:44749 every 8 connection(s)
I20260810 19:26:26.377494 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
I20260810 19:26:26.381141 31616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:26.382788 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 31363
I20260810 19:26:26.390690 31616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.60@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:26.408319 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.384948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.19.68.60:44749)
0810 19:26:26.385723 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.385738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.385788 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:26:26.387404 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.387424 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.387778 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.388105 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.390263 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.391576 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.393377 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.394534 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.394731 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.400230 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.400239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.400253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.400301 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.404032 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.404966 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.404973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.404979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.405173 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.405571 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.405575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.405577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.406123 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.406406 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.406907 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.407311 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260810 19:26:26.409775 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.393606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.19.68.62:42309)
0810 19:26:26.393803 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.393811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.394419 (+ 608us) server_negotiation.cc:407] Connection header received
0810 19:26:26.394641 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.394647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.394734 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.394834 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.396476 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.397751 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.399374 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.400147 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.400267 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.404259 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.404266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.404271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.404314 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.407050 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.407785 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.407791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.407795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.407932 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.408286 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.408291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.408295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.408609 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.408748 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.409209 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.409445 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:26:26.409838 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.393477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:42272)
0810 19:26:26.394271 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.394316 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.394478 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.394923 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.394931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.395797 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.396264 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.396281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.397888 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.397892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.399228 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.399240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.400321 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.400327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.400554 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.401236 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.401260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.404032 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.407209 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.407221 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.407228 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.407666 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.408037 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.408040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.408043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.408179 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.408723 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.408731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.408907 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.409070 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.409411 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":414,"threads_started":1}
I20260810 19:26:26.411211 31616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.60@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
I20260810 19:26:26.423328 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.411780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51726)
0810 19:26:26.411976 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.411991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.412067 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.412394 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.412398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.412705 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.412832 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.412839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.413855 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.413858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.414802 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.414813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.415632 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.415637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.415776 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.416266 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.416281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.418345 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.421234 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.421238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.421241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.421603 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.422029 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.422033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.422036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.422182 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.422662 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.422665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.422794 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.422900 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.423102 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 19:26:26.423430 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.412003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.19.68.61:44511)
0810 19:26:26.412162 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.412166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.412181 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:26:26.412231 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.412235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.412293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.412377 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.412950 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.413749 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.414971 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.415503 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.415616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.418552 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.418557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.418561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.418596 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.421074 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.421719 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.421723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.421725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.421868 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.422291 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.422295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.422297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.422574 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.422686 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.422984 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.423170 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:26:26.424263 31616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.60@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
I20260810 19:26:26.436363 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.424848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.0.0.1:39986)
0810 19:26:26.425038 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.425057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.425185 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.425469 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.425473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.425822 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.425966 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.425974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.426878 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.426883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.427714 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.427733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.428383 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.428388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.428554 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.429022 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.429044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.431124 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.434023 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.434028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.434033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.434381 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.434711 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.434716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.434720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.434884 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.435395 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.435399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.435559 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.435676 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.435934 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 19:26:26.436416 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.424943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.19.68.60:44749)
0810 19:26:26.425122 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.425128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.425144 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:26:26.425313 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.425316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.425366 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.425442 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.426090 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.426765 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.427841 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.428241 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.428344 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.431309 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.431313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.431316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.431343 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.433873 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.434494 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.434499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.434501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.434612 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.434986 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.434991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.434994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.435286 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.435413 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.435780 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.436083 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":70}
I20260810 19:26:26.441511 31616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Bootstrap starting.
I20260810 19:26:26.444667 31616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:26.445720 31616 log.cc:824] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:26.448386 31616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: No bootstrap required, opened a new log
I20260810 19:26:26.455540 31616 raft_consensus.cc:348] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } }
I20260810 19:26:26.455849 31616 raft_consensus.cc:374] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:26.455971 31616 raft_consensus.cc:729] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39095eeb1164695aad63abd8ff8405d, State: Initialized, Role: FOLLOWER
I20260810 19:26:26.456523 31616 consensus_queue.cc:260] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } }
I20260810 19:26:26.457404 31626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } } }
I20260810 19:26:26.457715 31626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:26.458386 31616 sys_catalog.cc:564] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:26:26.467689 31644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:26:26.467804 31644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:26:26.472054 31363 master_runner.cc:428] Detected that this master f39095eeb1164695aad63abd8ff8405d is joining an existing cluster
I20260810 19:26:26.472158 31363 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.68.60:44749
I20260810 19:26:26.493468 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.476137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.19.68.62:42309)
0810 19:26:26.476334 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.476340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.476962 (+ 622us) server_negotiation.cc:407] Connection header received
0810 19:26:26.477271 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.477277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.477361 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.477437 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.479049 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.480309 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.482500 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.483220 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.483360 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.485757 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.485764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.485769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.485813 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.489643 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.490468 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.490473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.490477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.490656 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.491173 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.491178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.491182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.491550 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.491696 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.492416 (+ 720us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.493077 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:26:26.494964 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.476153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:42278)
0810 19:26:26.476853 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.476876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.477071 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.477665 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.477670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.478490 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.478867 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.478881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.480482 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.480488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.482357 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.482374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.483411 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.483417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.483593 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.484317 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.484342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.485536 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.489829 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.489834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.489838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.490324 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.490803 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.490808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.490812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.491025 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.491787 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.491792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.492002 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.494018 (+ 2016us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.494448 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":380,"threads_started":1}
I20260810 19:26:26.503880 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.477061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.19.68.61:44511)
0810 19:26:26.477276 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.477282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.485730 (+ 8448us) server_negotiation.cc:407] Connection header received
0810 19:26:26.486022 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.486028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.486115 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.486192 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.487771 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.488934 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.491660 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.492345 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.492442 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.495086 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.495093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.495098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.495136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.499211 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.500428 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.500433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.500436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.500592 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.501156 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.501162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.501165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.501527 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.501669 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.503283 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.503512 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 19:26:26.504913 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.476946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:51738)
0810 19:26:26.485633 (+ 8687us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.485652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.485827 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.486419 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.486425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.487208 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.487589 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.487601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.489584 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.489589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.491438 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.491455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.492731 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.492737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.492890 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.493618 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.493641 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.494864 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.499783 (+ 4919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.499788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.499792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.500302 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.500760 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.500764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.500768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.500986 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.502277 (+ 1291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.502281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.502945 (+ 664us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.504146 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.504460 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8553,"thread_start_us":8373,"threads_started":1}
I20260810 19:26:26.507157 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.485926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.0.0.1:40002)
0810 19:26:26.492513 (+ 6587us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.492533 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.492651 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.492947 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.492951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.493679 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.494097 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.494111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.495508 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.495513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.496995 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.497010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.498297 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.498303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.498498 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.499063 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.499103 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.500259 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.504147 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.504151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.504154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.504670 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.505115 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.505119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.505122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.505268 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.506023 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.506028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.506176 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.506383 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.506659 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6454,"thread_start_us":6799,"threads_started":1}
I20260810 19:26:26.507165 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.485777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.19.68.60:44749)
0810 19:26:26.486754 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.486760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.492610 (+ 5850us) server_negotiation.cc:407] Connection header received
0810 19:26:26.492759 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.492763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.492835 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.492923 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.494242 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.495292 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.497164 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.497907 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.498010 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.500477 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.500483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.500487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.500522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.503994 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.504780 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.504784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.504787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.504964 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.505380 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.505384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.505391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.505770 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.506089 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.506512 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.506819 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":824}
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.62@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
I20260810 19:26:26.513967 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 30499
I20260810 19:26:26.519290 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.502481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.0.0.1:40004)
0810 19:26:26.502656 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.502674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.502777 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.503966 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.503971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.504445 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.504650 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.504662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.507262 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.507267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.508240 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.508256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.509619 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.509624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.509804 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.510343 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.510359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.512279 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.516418 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.516423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.516427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.516916 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.517466 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.517471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.517474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.517634 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.518263 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.518268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.518425 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.518609 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.518908 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:26:26.520416 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.502866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.19.68.60:44749)
0810 19:26:26.503573 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.503578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.503601 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:26:26.503686 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.503692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.503775 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.505205 (+ 1430us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.505645 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.507130 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.508394 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.509443 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.509570 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.512487 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.512494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.512499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.512534 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.516260 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.517126 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.517130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.517133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.517323 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.517746 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.517751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.517754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.518096 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.519679 (+ 1583us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.519797 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.519994 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":394,"threads_started":1}
I20260810 19:26:26.521720 30391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.68.60:44749: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f39095eeb1164695aad63abd8ff8405d"
member_type: NON_VOTER
last_known_addr {
host: "127.19.68.60"
port: 44749
}
attrs {
promote: true
}
}
dest_uuid: "57286387165c4738b948e530dd4a0bef"
cas_config_opid_index: 6
I20260810 19:26:26.522532 30391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } }
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.1@KRBTEST.COM
I20260810 19:26:26.525559 31574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:26.525698 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 19:26:26.528672 31574 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 19:26:26.529135 31359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 19:26:26.529366 31358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:26.531991 31358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f39095eeb1164695aad63abd8ff8405d (127.19.68.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.535379 31662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.515276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.19.68.1:34767)
0810 19:26:26.515947 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.515954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.515978 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:26:26.516031 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.516037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.516236 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.516379 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.517840 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.518998 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.520349 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.521033 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.521180 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.527042 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.527057 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.527074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.527115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.531134 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.532072 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.532081 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.532085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.532254 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.532718 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.532723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.532727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.533210 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.533532 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.534698 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.534968 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":373,"threads_started":1}
I20260810 19:26:26.538341 31574 raft_consensus.cc:858] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } }
W20260810 19:26:26.538635 31574 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } }
I20260810 19:26:26.538969 30717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:26.539608 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f39095eeb1164695aad63abd8ff8405d (127.19.68.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.544198 31491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.544530 31491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:26.549431 31358 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.68.60:44749
I20260810 19:26:26.550335 31360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.550657 31360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:26.551131 31359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.551415 31359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:26.567188 31574 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.569451 31363 master.cc:537] Master@127.19.68.60:44749 shutting down...
I20260810 19:26:26.581516 31674 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.581868 31674 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 19:26:26.582358 31626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57286387165c4738b948e530dd4a0bef. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.582593 31626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 19:26:26.582926 31668 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: NON_VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: true } } }
I20260810 19:26:26.583225 31668 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 19:26:26.584923 31359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.585299 31359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
W20260810 19:26:26.585942 30362 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 19:26:26.586200 31363 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 19:26:26.586658 31363 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 19:26:26.586858 31363 tablet_replica.cc:330] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: stopping tablet replica
I20260810 19:26:26.595459 31363 master.cc:552] Master@127.19.68.60:44749 shutdown complete.
I20260810 19:26:26.595557 31363 master_runner.cc:305] Clearing existing system tablet
I20260810 19:26:26.595961 31363 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:26.596061 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260810 19:26:26.599200 30362 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef -> Peer f39095eeb1164695aad63abd8ff8405d (127.19.68.60:44749): Couldn't send request to peer f39095eeb1164695aad63abd8ff8405d. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260810 19:26:26.600092 31363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:26:26.601794 31681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:26.602222 31363 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:26.602632 31363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 19:26:26.602792 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "f39095eeb1164695aad63abd8ff8405d"
format_stamp: "Formatted at 2026-08-10 19:26:26 on dist-test-slave-2j7r"
server_key: "228550bf7c9fe0d782bcbef22465b769"
server_key_iv: "71aca68384281d84da1827ed2a6583a2"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.607534 31363 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:26.614532 31363 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 19:26:26.614795 31363 log.cc:1196] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 19:26:26.615432 31363 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Deleting consensus metadata
I20260810 19:26:26.615761 31363 master_runner.cc:315] Copying system tablet from 127.19.68.62:42309
I20260810 19:26:26.618467 31363 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.68.62:42309
I20260810 19:26:26.635656 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.619393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.19.68.62:42309)
0810 19:26:26.619629 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:26.619635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:26.621615 (+ 1980us) server_negotiation.cc:407] Connection header received
0810 19:26:26.621917 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:26.621924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:26.622021 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:26.622126 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:26.623167 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.624278 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.626076 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.626741 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.626871 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.629163 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:26.629169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:26.629175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:26.629215 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:26.632660 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.633508 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.633515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.633519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.633677 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:26.634100 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:26.634105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:26.634109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:26.634466 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:26.634606 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:26.635059 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 19:26:26.635302 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 19:26:26.636083 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:26.619303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:39856)
0810 19:26:26.621491 (+ 2188us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:26.621511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:26.621734 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:26.622169 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:26.622174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:26.622701 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:26.623003 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.623017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.624429 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.624434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.625907 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:26.625923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:26.626909 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:26.626915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:26.627111 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:26.627817 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:26.627840 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:26.628919 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:26.632808 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.632814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.632818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.633395 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.633798 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:26.633803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:26.633806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:26.633980 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:26.634586 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:26.634591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:26.634798 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 19:26:26.634947 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 19:26:26.635274 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2031,"thread_start_us":1920,"threads_started":1}
I20260810 19:26:26.636708 30418 tablet_copy_service.cc:140] P 57286387165c4738b948e530dd4a0bef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f39095eeb1164695aad63abd8ff8405d ({username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:39856)
I20260810 19:26:26.636884 30418 tablet_copy_service.cc:161] P 57286387165c4738b948e530dd4a0bef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f39095eeb1164695aad63abd8ff8405d at {username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:39856: session id = f39095eeb1164695aad63abd8ff8405d-00000000000000000000000000000000
I20260810 19:26:26.640578 30418 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 19:26:26.642481 31363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 19:26:26.649238 31363 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: tablet copy: Starting download of 0 data blocks...
I20260810 19:26:26.649559 31363 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: tablet copy: Starting download of 1 WAL segments...
I20260810 19:26:26.651947 31363 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 19:26:26.656342 30418 tablet_copy_service.cc:342] P 57286387165c4738b948e530dd4a0bef: Request end of tablet copy session f39095eeb1164695aad63abd8ff8405d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:39856
I20260810 19:26:26.656495 30418 tablet_copy_service.cc:433] P 57286387165c4738b948e530dd4a0bef: ending tablet copy session f39095eeb1164695aad63abd8ff8405d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f39095eeb1164695aad63abd8ff8405d
I20260810 19:26:26.669827 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:26.672797 31694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:26.673430 31696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:26.681958 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 30499
W20260810 19:26:26.685999 31699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:26.692812 31363 server_base.cc:1034] running on GCE node
I20260810 19:26:26.693216 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:26.693846 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:26.695044 31363 hybrid_clock.cc:648] HybridClock initialized: now 1786389986695001 us; error 34 us; skew 500 ppm
I20260810 19:26:26.695282 31363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:26 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389986, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:26.697057 31363 init.cc:377] Logged in from keytab as oryx/127.19.68.60@KRBTEST.COM (short username oryx)
I20260810 19:26:26.702654 31363 webserver.cc:460] Webserver started at http://127.19.68.60:46469/ using document root <none> and password file <none>
I20260810 19:26:26.702935 31363 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:26.703038 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:26.705415 31363 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:26:26.707042 31712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:26.707854 31363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 19:26:26.707978 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "f39095eeb1164695aad63abd8ff8405d"
format_stamp: "Formatted at 2026-08-10 19:26:26 on dist-test-slave-2j7r"
server_key: "228550bf7c9fe0d782bcbef22465b769"
server_key_iv: "71aca68384281d84da1827ed2a6583a2"
server_key_version: "encryptionkey@0"
I20260810 19:26:26.708168 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:26.716445 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:34767
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:42309
--webserver_port=45241
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:26:26.725898 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:26.729596 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:26.754019 31363 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.60:44749
I20260810 19:26:26.754055 31802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.60:44749 every 8 connection(s)
I20260810 19:26:26.755002 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
I20260810 19:26:26.761206 31803 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:26:26.765648 31803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Bootstrap starting.
I20260810 19:26:26.780483 31803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:26:26.780959 31803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Bootstrap complete.
I20260810 19:26:26.781894 31803 raft_consensus.cc:348] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } }
I20260810 19:26:26.782145 31803 raft_consensus.cc:729] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f39095eeb1164695aad63abd8ff8405d, State: Initialized, Role: LEARNER
I20260810 19:26:26.782451 31803 consensus_queue.cc:260] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } }
I20260810 19:26:26.789583 31806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:26.790060 31803 sys_catalog.cc:564] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:26.790271 31806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:26.804775 31826 catalog_manager.cc:1259] Loaded cluster ID: a0a171afd5874fdaafd1137337b2a4a2
I20260810 19:26:26.804886 31826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: loading cluster ID for follower catalog manager: success
I20260810 19:26:26.808516 31826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: acquiring CA information for follower catalog manager: success
I20260810 19:26:26.810889 31826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 19:26:27.075744 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:27.076030 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:27.076130 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:27.083421 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:27.083523 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:27.083647 31721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:27.093641 31721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:34767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=45241
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:27.094892 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:27.096130 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:27.109429 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:27.109318 31841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:27.109596 31837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:27.110314 31721 server_base.cc:1034] running on GCE node
I20260810 19:26:27.110934 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:27.111804 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:27.113015 31721 hybrid_clock.cc:648] HybridClock initialized: now 1786389987112964 us; error 35 us; skew 500 ppm
I20260810 19:26:27.113354 31721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:27.116590 31721 init.cc:377] Logged in from keytab as oryx/127.19.68.1@KRBTEST.COM (short username oryx)
I20260810 19:26:27.118158 31721 webserver.cc:460] Webserver started at http://127.19.68.1:45241/ using document root <none> and password file <none>
I20260810 19:26:27.118829 31721 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:27.118973 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:27.124514 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:26:27.127281 31860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.128561 31721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:27.128710 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "0595e8ff3ebc42a39d0323a61440eb12"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "7eb156ada7e1cc2628fc8264bcb36374"
server_key_iv: "e0c658da202752ff797af0e68e2d3baf"
server_key_version: "encryptionkey@0"
I20260810 19:26:27.129760 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:27.160136 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:27.163995 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:27.164248 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:27.165225 31721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:27.166633 31721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:27.166723 31721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.166826 31721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:27.166903 31721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.198971 31721 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34767
I20260810 19:26:27.199065 32066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34767 every 8 connection(s)
I20260810 19:26:27.199918 31855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:27.200302 31855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:27.200630 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:27.206911 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 31721
I20260810 19:26:27.207309 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 30735
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.2@KRBTEST.COM
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
I20260810 19:26:27.235544 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.200334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.0.0.1:59214)
0810 19:26:27.216691 (+ 16357us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.216709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.216852 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.218873 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.218878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.218913 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:27.219158 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.219169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.221166 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.221171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.223158 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.223174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.234683 (+ 11509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.234690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.234906 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.234944 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.235036 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.235192 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16224,"thread_start_us":16158,"threads_started":1}
I20260810 19:26:27.235589 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.200654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.19.68.60:44749)
0810 19:26:27.201275 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.201281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.216805 (+ 15524us) server_negotiation.cc:407] Connection header received
0810 19:26:27.218209 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.218216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.218340 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.218488 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:27.219341 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.220938 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.232899 (+ 11961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.234482 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.234609 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.235013 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.235067 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.235197 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":288,"threads_started":1}
I20260810 19:26:27.236778 31754 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260810 19:26:27.237370 31806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57286387165c4738b948e530dd4a0bef. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: NON_VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: true } } }
I20260810 19:26:27.237634 31806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: LEARNER
I20260810 19:26:27.238394 31855 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: attempting to promote NON_VOTER f39095eeb1164695aad63abd8ff8405d to VOTER
I20260810 19:26:27.239145 31855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } }
I20260810 19:26:27.241914 31754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 LEARNER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 19:26:27.242291 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Refusing update from remote peer 57286387165c4738b948e530dd4a0bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 19:26:27.242650 32068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 19:26:27.243072 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.208936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.19.68.2:33205)
0810 19:26:27.215170 (+ 6234us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.215181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.216465 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 19:26:27.217288 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.217298 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.217605 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.217782 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.220174 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.221861 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.223624 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.224455 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.224573 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.230658 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.230674 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.230690 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.230741 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.238557 (+ 7816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.239495 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.239504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.239507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.239684 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.240182 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.240187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.240190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.240635 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.242216 (+ 1581us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.242383 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.242668 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6039,"thread_start_us":5876,"threads_started":1}
I20260810 19:26:27.242990 31855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 19:26:27.244886 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.209713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51409 (local address 127.19.68.60:44749)
0810 19:26:27.218666 (+ 8953us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.218672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.222508 (+ 3836us) server_negotiation.cc:407] Connection header received
0810 19:26:27.222563 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.222567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.222652 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.222751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.225843 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.227058 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.229791 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.230449 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.230542 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.235354 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.235361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.235366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.235414 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.239281 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.240287 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.240292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.240296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.240452 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.241021 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.241026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.241030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.241409 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.241548 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.244269 (+ 2721us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.244507 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8802,"thread_start_us":8974,"threads_started":1}
I20260810 19:26:27.245409 30939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:27.245523 32068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f39095eeb1164695aad63abd8ff8405d (127.19.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.246474 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.209028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.19.68.1:51409)
0810 19:26:27.219377 (+ 10349us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.219434 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.222066 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.223867 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.223879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.224700 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.225605 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.225647 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.227288 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.227294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.229612 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.229633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.230906 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.230914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.231226 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.231938 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.231971 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.235005 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.239461 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.239471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.239490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.240127 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.240608 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.240613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.240617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.240886 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.241783 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.241793 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.242201 (+ 408us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.242525 (+ 324us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.243079 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9922,"thread_start_us":14091,"threads_started":1}
I20260810 19:26:27.248422 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f39095eeb1164695aad63abd8ff8405d (127.19.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.249725 32073 heartbeater.cc:347] Connected to a master server at 127.19.68.60:44749
I20260810 19:26:27.249966 32073 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.250820 32073 heartbeater.cc:511] Master 127.19.68.60:44749 requested a full tablet report, sending...
I20260810 19:26:27.253539 31739 ts_manager.cc:194] Registered new tserver with Master: 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767)
I20260810 19:26:27.255219 31754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f39095eeb1164695aad63abd8ff8405d (127.19.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.259392 31806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.259635 31806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:27.261716 32067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.262080 32067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:27.262336 31855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.262602 31855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:27.263315 32101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.263666 32101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:27.263676 32068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57286387165c4738b948e530dd4a0bef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:27.264684 32068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef [sys.catalog]: This master's current role is: LEADER
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
I20260810 19:26:27.272080 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.228064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:33775)
0810 19:26:27.246863 (+ 18799us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.246882 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.246982 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.255664 (+ 8682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.255668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.256123 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.256312 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.256324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.258181 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.258187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.259505 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.259519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.261820 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.261827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.262009 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.262637 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.262661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.265537 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.269472 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.269478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.269481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.269983 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.270343 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.270348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.270352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.270471 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.270964 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.270969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.271131 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.271282 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.271626 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18683,"thread_start_us":19027,"threads_started":1}
I20260810 19:26:27.272096 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.224520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:33775 (local address 127.19.68.62:42309)
0810 19:26:27.255263 (+ 30743us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.255269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.255293 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:26:27.255340 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.255345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.255417 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.255515 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.256603 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.258026 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.259676 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.261197 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.261322 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.265729 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.265736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.265741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.265789 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.269199 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.270097 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.270102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.270104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.270229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.270570 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.270574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.270576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.270840 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.270974 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.271416 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.271621 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30602,"thread_start_us":30288,"threads_started":1}
I20260810 19:26:27.273056 32078 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:27.273228 32078 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.273684 32078 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:27.275189 30391 ts_manager.cc:194] Re-registered known tserver with Master: 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767)
I20260810 19:26:27.276201 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:33775
I20260810 19:26:27.276744 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.247569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.19.68.1:40293)
0810 19:26:27.247740 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.247757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.247859 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.259631 (+ 11772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.259636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.260243 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.260544 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.260556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.264074 (+ 3518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.264079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.265606 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.265622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.266632 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.266638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.266819 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.267367 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.267389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.269935 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.274153 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.274158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.274162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.274636 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.275033 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.275038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.275041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.275207 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.275825 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.275830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.275979 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.276123 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.276395 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 19:26:27.276791 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.247710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40293 (local address 127.19.68.61:44511)
0810 19:26:27.258658 (+ 10948us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.258665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.258692 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:26:27.258751 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.258758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.258845 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.260717 (+ 1872us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.260952 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.262091 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.265736 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.266428 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.267116 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.270120 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.270126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.270131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.270168 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.274018 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.274768 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.274773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.274776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.274919 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.275341 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.275346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.275350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.275684 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.275840 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.276212 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.276430 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10807,"thread_start_us":12167,"threads_started":1}
I20260810 19:26:27.277783 32090 heartbeater.cc:347] Connected to a master server at 127.19.68.61:44511
I20260810 19:26:27.277933 32090 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.278324 32090 heartbeater.cc:511] Master 127.19.68.61:44511 requested a full tablet report, sending...
I20260810 19:26:27.279814 31426 ts_manager.cc:194] Registered new tserver with Master: 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767)
I20260810 19:26:27.351320 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 30735
I20260810 19:26:27.373014 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:33205
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:42309
--webserver_port=43641
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:27.716905 32106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:27.717247 32106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:27.717412 32106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:27.724637 32106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:27.724733 32106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:27.724875 32106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:27.734750 32106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:33205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=43641
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:27.736078 32106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:27.737457 32106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:27.750065 32114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:27.750171 32115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:27.750118 32118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:27.751376 32106 server_base.cc:1034] running on GCE node
I20260810 19:26:27.751957 32106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:27.752920 32106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:27.754134 32106 hybrid_clock.cc:648] HybridClock initialized: now 1786389987754079 us; error 50 us; skew 500 ppm
I20260810 19:26:27.754436 32106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:27.757692 32106 init.cc:377] Logged in from keytab as oryx/127.19.68.2@KRBTEST.COM (short username oryx)
I20260810 19:26:27.759284 32106 webserver.cc:460] Webserver started at http://127.19.68.2:43641/ using document root <none> and password file <none>
I20260810 19:26:27.760109 32106 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:27.760262 32106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:27.766378 32106 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:26:27.769057 32131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.770363 32106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:27.770511 32106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "de73a592a4e94614a914a184aceb2a87"
format_stamp: "Formatted at 2026-08-10 19:26:24 on dist-test-slave-2j7r"
server_key: "8e12d866bfb62f28b39757a06f1b32d0"
server_key_iv: "d1e2c32950e54774590972093ae32290"
server_key_version: "encryptionkey@0"
I20260810 19:26:27.771476 32106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:27.791676 32106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:27.795495 32106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:27.795737 32106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:27.796622 32106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:27.798027 32106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:27.798122 32106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.798229 32106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:27.798300 32106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:27.830502 32106 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:33205
I20260810 19:26:27.830642 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:33205 every 8 connection(s)
I20260810 19:26:27.831921 32106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:27.838493 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 32106
I20260810 19:26:27.838801 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 30958
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389987, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:27.860646 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.836674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:60583 (local address 127.19.68.61:44511)
0810 19:26:27.837586 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.837592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.837610 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:26:27.838760 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.838767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.838918 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.839100 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.841500 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.842746 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.845439 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.846044 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.846149 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.851530 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.851537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.851542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.851583 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.856115 (+ 4532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.857427 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.857433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.857436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.857561 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.858060 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.858065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.858068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.858460 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.858608 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.860042 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.860269 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":689,"threads_started":1}
Aug 10 19:26:27 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.3@KRBTEST.COM
I20260810 19:26:27.862701 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.837614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:48407 (local address 127.19.68.60:44749)
0810 19:26:27.838086 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.838092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.845687 (+ 7595us) server_negotiation.cc:407] Connection header received
0810 19:26:27.846109 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.846115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.846197 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.846293 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.848309 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.849289 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.850729 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.851352 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.851477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.855535 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.855540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.855543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.855575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.859153 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.860601 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.860606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.860609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.860765 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.861218 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.861223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.861227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.861585 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.861713 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.862127 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.862360 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":255,"threads_started":1}
I20260810 19:26:27.863636 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.836509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.19.68.2:60583)
0810 19:26:27.837141 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.837157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.838234 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.839353 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.839364 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.840301 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.841275 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.841301 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.842941 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.842947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.845229 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.845245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.846416 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.846424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.846719 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.847489 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.847517 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.851271 (+ 3754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.856505 (+ 5234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.856514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.856530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.857304 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.857675 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.857679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.857683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.857916 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.858614 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.858623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.859093 (+ 470us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.859451 (+ 358us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.862250 (+ 2799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":20,"thread_start_us":291,"threads_started":1}
I20260810 19:26:27.864054 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.837177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.19.68.2:48407)
0810 19:26:27.845592 (+ 8415us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.845608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.845932 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.846521 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.846526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.847075 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.848123 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.848135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.849473 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.849478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.850581 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.850594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.851724 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.851730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.851910 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.852476 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.852499 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.855352 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.859310 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.859315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.859318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.860457 (+ 1139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.860898 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.860903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.860907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.861065 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.861772 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.861775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.861877 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.861977 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.863572 (+ 1595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8301,"mutex_wait_us":1303,"spinlock_wait_cycles":15744,"thread_start_us":8530,"threads_started":1}
I20260810 19:26:27.865439 32324 heartbeater.cc:347] Connected to a master server at 127.19.68.60:44749
I20260810 19:26:27.865586 32328 heartbeater.cc:347] Connected to a master server at 127.19.68.61:44511
I20260810 19:26:27.865788 32328 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.866367 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.835262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:54119)
0810 19:26:27.836011 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:27.836070 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:27.838224 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:27.839354 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:27.839365 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:27.840300 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:27.849026 (+ 8726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.849040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.850796 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.850801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.852623 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:27.852639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.854479 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.854486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:27.854705 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.856080 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:27.856102 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:27.859343 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:27.862973 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.862978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.862981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.863396 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.864315 (+ 919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:27.864321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:27.864324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:27.864476 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:27.865280 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:27.865285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:27.865466 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 19:26:27.865630 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 19:26:27.865937 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":6704,"spinlock_wait_cycles":33536,"thread_start_us":263,"threads_started":1}
I20260810 19:26:27.866612 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.834774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:54119 (local address 127.19.68.62:42309)
0810 19:26:27.835243 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.835250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.836183 (+ 933us) server_negotiation.cc:407] Connection header received
0810 19:26:27.838656 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.838663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.838771 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.838879 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.849241 (+ 10362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.850250 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.852774 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.853326 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.853414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.859525 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.859530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.859540 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.859576 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.862819 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.863586 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.863591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.863595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.863779 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.864684 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.864689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.864692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.865157 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.865328 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.865991 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.866223 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
I20260810 19:26:27.865785 32324 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.867296 32328 heartbeater.cc:511] Master 127.19.68.61:44511 requested a full tablet report, sending...
I20260810 19:26:27.867411 32326 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:27.867506 32324 heartbeater.cc:511] Master 127.19.68.60:44749 requested a full tablet report, sending...
I20260810 19:26:27.867560 32326 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:27.868031 32326 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:27.869158 30391 ts_manager.cc:194] Re-registered known tserver with Master: de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:27.869298 31739 ts_manager.cc:194] Registered new tserver with Master: de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:27.869336 31426 ts_manager.cc:194] Registered new tserver with Master: de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:27.870186 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:54119
I20260810 19:26:27.871336 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:27.840183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.19.68.3:42349)
0810 19:26:27.848717 (+ 8534us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:27.848725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:27.848762 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:26:27.848830 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:27.848838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:27.849108 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:27.849262 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:27.851484 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.852906 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.855168 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:27.856041 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:27.856179 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:27.862463 (+ 6284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:27.862478 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:27.862495 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:27.862536 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:27.866714 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.867839 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.867847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.867851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.868013 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:27.868951 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:27.868956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:27.868959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:27.869577 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:27.869870 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:27.870445 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 19:26:27.871023 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8357,"thread_start_us":8199,"threads_started":1}
I20260810 19:26:27.872591 31182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:27.956369 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 30958
I20260810 19:26:27.980283 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:42349
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:42309
--webserver_port=40701
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:26:28.279145 32078 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
W20260810 19:26:28.309420 32352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:28.309679 32352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:28.309813 32352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:28.316835 32352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:28.316936 32352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:28.317050 32352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:28.326364 32352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:42349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=40701
--tserver_master_addrs=127.19.68.62:42309,127.19.68.61:44511,127.19.68.60:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:28.327533 32352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:28.328795 32352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:28.341692 32361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:28.341660 32367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:28.341722 32362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:28.342223 32352 server_base.cc:1034] running on GCE node
I20260810 19:26:28.342864 32352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:28.343698 32352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:28.344894 32352 hybrid_clock.cc:648] HybridClock initialized: now 1786389988344834 us; error 53 us; skew 500 ppm
I20260810 19:26:28.345218 32352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:28 dist-test-slave-2j7r krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389988, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:28.348282 32352 init.cc:377] Logged in from keytab as oryx/127.19.68.3@KRBTEST.COM (short username oryx)
I20260810 19:26:28.349624 32352 webserver.cc:460] Webserver started at http://127.19.68.3:40701/ using document root <none> and password file <none>
I20260810 19:26:28.350250 32352 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:28.350378 32352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:28.355396 32352 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:26:28.357969 32382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:28.359194 32352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:28.359335 32352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "ec40d07ba1ab434988bd05c03777fcce"
format_stamp: "Formatted at 2026-08-10 19:26:25 on dist-test-slave-2j7r"
server_key: "0c4c16c8ce22cd504db951c85f220512"
server_key_iv: "00cdde44220a924c271697a95bf57978"
server_key_version: "encryptionkey@0"
I20260810 19:26:28.360271 32352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:28.389977 32352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:28.393774 32352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:28.394039 32352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:28.394982 32352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:28.396293 32352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:28.396389 32352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:28.396492 32352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:28.396564 32352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:28.426771 32352 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42349
I20260810 19:26:28.426819 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42349 every 8 connection(s)
I20260810 19:26:28.428232 32352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:28.434705 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 32352
Aug 10 19:26:28 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389988, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.60@KRBTEST.COM
Aug 10 19:26:28 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389988, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.61@KRBTEST.COM
I20260810 19:26:28.456295 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.431084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:39339 (local address 127.19.68.60:44749)
0810 19:26:28.431568 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.431572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.432634 (+ 1062us) server_negotiation.cc:407] Connection header received
0810 19:26:28.435846 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.435854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.435964 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.436050 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.438667 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.440063 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.442701 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.443389 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.443489 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.448727 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.448734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.448739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.448777 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.452397 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.453451 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.453457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.453460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.453606 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.454107 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.454113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.454116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.454474 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.454607 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.455666 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.455906 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":224,"threads_started":1}
Aug 10 19:26:28 dist-test-slave-2j7r krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389988, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:28.458627 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.431702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.19.68.3:39339)
0810 19:26:28.432449 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.432509 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.435310 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.436538 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.436551 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.437578 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:28.438443 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.438470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.440297 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.440302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.442513 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.442530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.443916 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.443924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.444246 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.445058 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:28.445124 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:28.448472 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:28.452579 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.452588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.452603 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.453291 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.453730 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.453734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.453738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.453978 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.454710 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:28.454718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:28.455153 (+ 435us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.456604 (+ 1451us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.457124 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":281,"threads_started":1}
I20260810 19:26:28.460531 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.432900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.19.68.3:37709)
0810 19:26:28.433508 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.433526 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.435589 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.445387 (+ 9798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.445395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.446104 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:28.446498 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.446511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.447742 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.447748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.449297 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.449312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.450256 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.450262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.450421 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.450904 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:28.450928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:28.453477 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:28.457707 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.457716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.457719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.458312 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.458938 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.458943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.458946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.459089 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.459676 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:28.459680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:28.459839 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.459964 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.460217 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":635,"threads_started":1}
I20260810 19:26:28.460711 32561 heartbeater.cc:347] Connected to a master server at 127.19.68.60:44749
I20260810 19:26:28.461009 32561 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:28.461932 32561 heartbeater.cc:511] Master 127.19.68.60:44749 requested a full tablet report, sending...
I20260810 19:26:28.462191 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.433067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:37709 (local address 127.19.68.61:44511)
0810 19:26:28.445012 (+ 11945us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.445018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.445038 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:26:28.445124 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.445130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.445209 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.446667 (+ 1458us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.446869 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.447518 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.449488 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.450051 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.450677 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.453718 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.453725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.453729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.453772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.457558 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.458480 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.458484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.458487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.458646 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.459211 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.459215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.459218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.459541 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.461498 (+ 1957us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.461633 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.461853 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11821,"thread_start_us":11657,"threads_started":1}
I20260810 19:26:28.463075 32563 heartbeater.cc:347] Connected to a master server at 127.19.68.61:44511
I20260810 19:26:28.463204 32563 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:28.463676 32563 heartbeater.cc:511] Master 127.19.68.61:44511 requested a full tablet report, sending...
I20260810 19:26:28.464814 31739 ts_manager.cc:194] Registered new tserver with Master: ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349)
I20260810 19:26:28.464833 31426 ts_manager.cc:194] Registered new tserver with Master: ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349)
I20260810 19:26:28.467418 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.433904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:56657)
0810 19:26:28.446593 (+ 12689us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.446617 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.446737 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.448801 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.448806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.449749 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:28.450079 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.450092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.451737 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.451743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.453390 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.453407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.454473 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.454480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.454656 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.455181 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:28.455202 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:28.458395 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:28.463686 (+ 5291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.463691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.463695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.464156 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.465682 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:28.465688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:28.465691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:28.465857 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:28.466505 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:28.466510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:28.466658 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.466776 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.467036 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12543,"thread_start_us":13641,"threads_started":1}
I20260810 19:26:28.468650 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.434037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:56657 (local address 127.19.68.62:42309)
0810 19:26:28.448450 (+ 14413us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.448457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.448479 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:26:28.448542 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.448548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.448624 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.450260 (+ 1636us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.450432 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.451566 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.453597 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.454256 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.454960 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.458623 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.458630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.458635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.458684 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.463540 (+ 4856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.465377 (+ 1837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.465382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.465385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.465552 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.466023 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.466027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.466031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.466362 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.467927 (+ 1565us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.468062 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.468273 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14285,"thread_start_us":14172,"threads_started":1}
I20260810 19:26:28.469763 32562 heartbeater.cc:347] Connected to a master server at 127.19.68.62:42309
I20260810 19:26:28.469898 32562 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:28.470307 32562 heartbeater.cc:511] Master 127.19.68.62:42309 requested a full tablet report, sending...
I20260810 19:26:28.471313 30391 ts_manager.cc:194] Re-registered known tserver with Master: ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349)
I20260810 19:26:28.472519 30391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:56657
I20260810 19:26:28.483314 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.463489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.19.68.62:42309)
0810 19:26:28.467114 (+ 3625us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.467120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.467138 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:26:28.467182 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.467187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.467265 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.469708 (+ 2443us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.469908 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.470852 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.472828 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.473497 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.474215 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.477022 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.477030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.477036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.477076 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.480006 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.480770 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.480775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.480779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.480928 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.481357 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.481362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.481365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.481716 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.481932 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.482677 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.482907 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3521,"thread_start_us":3407,"threads_started":1}
I20260810 19:26:28.485965 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.466580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.19.68.61:44511)
0810 19:26:28.466753 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.466759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.467572 (+ 813us) server_negotiation.cc:407] Connection header received
0810 19:26:28.467816 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.467821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.467924 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.467993 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.470544 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.471542 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.473385 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.473985 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.474080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.479396 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.479402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.479406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.479445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.482696 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.483723 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.483728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.483731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.483889 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.484596 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.484601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.484605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.484873 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.485170 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.485508 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.485712 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:26:28.492529 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.467551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.19.68.60:44749)
0810 19:26:28.467710 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.467715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.477720 (+ 10005us) server_negotiation.cc:407] Connection header received
0810 19:26:28.477900 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.477905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.477989 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.478088 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.479806 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.480681 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.482503 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.483105 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.483332 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.486723 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.486727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.486730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.486762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.489923 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.490566 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.490571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.490575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.490747 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.491059 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.491064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.491068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.491432 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.491704 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.491963 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.492189 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:26:28.487952 30391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:28.496868 30391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:28.532212 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.514935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33205 (local address 127.0.0.1:49106)
0810 19:26:28.522074 (+ 7139us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.522093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.522228 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.523237 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.523242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.523270 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.523640 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.523651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.525660 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.525666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.527955 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.527972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.531388 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.531410 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.531597 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.531632 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.531713 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.531844 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7009,"thread_start_us":6857,"threads_started":1}
I20260810 19:26:28.532684 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.515555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.19.68.2:33205)
0810 19:26:28.521597 (+ 6042us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.521607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.522354 (+ 747us) server_negotiation.cc:407] Connection header received
0810 19:26:28.522495 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.522504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.522786 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.522993 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.523830 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.525473 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.528153 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.529661 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.529788 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.530030 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.531953 (+ 1923us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.532160 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5856,"thread_start_us":5711,"threads_started":1}
I20260810 19:26:28.539523 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.514384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42349 (local address 127.0.0.1:49938)
0810 19:26:28.516243 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.516267 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.516405 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.531003 (+ 14598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.531008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.531032 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.531500 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.531512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.534388 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.534393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.536570 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.536586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.538648 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.538654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.538883 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.538920 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.539007 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.539131 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725,"thread_start_us":1557,"threads_started":1}
I20260810 19:26:28.540324 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.514862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.19.68.3:42349)
0810 19:26:28.530541 (+ 15679us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.530547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.530573 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:26:28.530644 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.530649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.530820 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.530989 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.532472 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.534226 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.536745 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.538198 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.538306 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.538505 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.539854 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.540016 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15507,"thread_start_us":15329,"threads_started":1}
I20260810 19:26:28.543972 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.522160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34767 (local address 127.0.0.1:43448)
0810 19:26:28.533325 (+ 11165us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.533342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.533433 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.535905 (+ 2472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.535911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.535944 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.536186 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.536199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.538585 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.538592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.540876 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.540893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.543035 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.543042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.543251 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.543289 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.543394 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.543542 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11052,"mutex_wait_us":362,"thread_start_us":16683,"threads_started":1}
I20260810 19:26:28.544123 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 89c23633e3e546c68f43baa93efcd352 (DEFAULT_TABLE table=test-table [id=696675f089244c8e9a634e97cc1e9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:28.544119 32210 tablet_service.cc:1467] Processing CreateTablet for tablet 89c23633e3e546c68f43baa93efcd352 (DEFAULT_TABLE table=test-table [id=696675f089244c8e9a634e97cc1e9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:28.544827 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.522525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.19.68.1:34767)
0810 19:26:28.535025 (+ 12500us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.535035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.535068 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:26:28.535179 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.535189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.535432 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.536353 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.536634 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.538383 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.541102 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.542556 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.542692 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.542928 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.544227 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.544441 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12317,"thread_start_us":11968,"threads_started":1}
I20260810 19:26:28.545684 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c23633e3e546c68f43baa93efcd352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:28.545732 32210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c23633e3e546c68f43baa93efcd352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:28.553864 31954 tablet_service.cc:1467] Processing CreateTablet for tablet 89c23633e3e546c68f43baa93efcd352 (DEFAULT_TABLE table=test-table [id=696675f089244c8e9a634e97cc1e9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:28.555464 31954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c23633e3e546c68f43baa93efcd352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:28.561797 32626 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Bootstrap starting.
I20260810 19:26:28.564736 32627 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Bootstrap starting.
I20260810 19:26:28.565591 32626 tablet_bootstrap.cc:654] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:28.566880 32626 log.cc:824] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:28.568069 32627 tablet_bootstrap.cc:654] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:28.569357 32627 log.cc:824] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:28.570277 32626 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: No bootstrap required, opened a new log
I20260810 19:26:28.570516 32626 ts_tablet_manager.cc:1397] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:26:28.572101 32633 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Bootstrap starting.
I20260810 19:26:28.572520 32627 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: No bootstrap required, opened a new log
I20260810 19:26:28.572770 32627 ts_tablet_manager.cc:1397] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 19:26:28.576478 32633 tablet_bootstrap.cc:654] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:28.577541 32633 log.cc:824] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:28.580086 32633 tablet_bootstrap.cc:492] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: No bootstrap required, opened a new log
I20260810 19:26:28.580334 32633 ts_tablet_manager.cc:1397] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 19:26:28.581012 32626 raft_consensus.cc:348] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.581452 32626 raft_consensus.cc:374] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:28.581543 32626 raft_consensus.cc:729] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40d07ba1ab434988bd05c03777fcce, State: Initialized, Role: FOLLOWER
I20260810 19:26:28.582280 32626 consensus_queue.cc:260] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.582963 32627 raft_consensus.cc:348] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.583369 32627 raft_consensus.cc:374] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:28.583467 32627 raft_consensus.cc:729] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de73a592a4e94614a914a184aceb2a87, State: Initialized, Role: FOLLOWER
I20260810 19:26:28.584117 32627 consensus_queue.cc:260] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.589499 32633 raft_consensus.cc:348] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.590037 32633 raft_consensus.cc:374] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:28.590209 32633 raft_consensus.cc:729] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0595e8ff3ebc42a39d0323a61440eb12, State: Initialized, Role: FOLLOWER
I20260810 19:26:28.590392 32627 ts_tablet_manager.cc:1428] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260810 19:26:28.590387 32626 ts_tablet_manager.cc:1428] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260810 19:26:28.590724 32326 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
I20260810 19:26:28.591207 32633 consensus_queue.cc:260] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.592372 32562 heartbeater.cc:503] Master 127.19.68.62:42309 was elected leader, sending a full tablet report...
I20260810 19:26:28.600425 32633 ts_tablet_manager.cc:1428] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260810 19:26:28.606137 32639 raft_consensus.cc:482] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:28.606384 32639 raft_consensus.cc:504] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.607858 32639 leader_election.cc:288] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349), de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:28.609969 32638 raft_consensus.cc:482] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:28.610311 32638 raft_consensus.cc:504] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.612393 32638 leader_election.cc:288] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349), 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767)
I20260810 19:26:28.615937 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.608153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:45899 (local address 127.19.68.3:42349)
0810 19:26:28.608317 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.608323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.609169 (+ 846us) server_negotiation.cc:407] Connection header received
0810 19:26:28.609472 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.609478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.609559 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.609642 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.610549 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.611334 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.613591 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.614765 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.614861 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.614974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.615523 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.615661 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:26:28.616961 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.608184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42349 (local address 127.19.68.1:45899)
0810 19:26:28.609048 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.609072 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.609305 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.609862 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.609866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.609889 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.610352 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.610369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.611460 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.611465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.613406 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.613423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.615165 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.615171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.615344 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.615377 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.616375 (+ 998us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.616517 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":3,"thread_start_us":487,"threads_started":1}
I20260810 19:26:28.617672 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.608647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:52945 (local address 127.19.68.2:33205)
0810 19:26:28.608901 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.608908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.609548 (+ 640us) server_negotiation.cc:407] Connection header received
0810 19:26:28.609768 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.609773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.609843 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.609928 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.610767 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.611898 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.614762 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.616246 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.616378 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.616492 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.617221 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.617361 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 19:26:28.618443 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.608292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33205 (local address 127.19.68.1:52945)
0810 19:26:28.609446 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.609465 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.609647 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.609993 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.610003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.610033 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.610544 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.610559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.612112 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.612123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.614519 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.614538 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.616753 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.616761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.616986 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.617026 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.617925 (+ 899us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.618064 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":23,"thread_start_us":862,"threads_started":1}
I20260810 19:26:28.618217 32485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "0595e8ff3ebc42a39d0323a61440eb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40d07ba1ab434988bd05c03777fcce" is_pre_election: true
I20260810 19:26:28.618855 32485 raft_consensus.cc:2454] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0595e8ff3ebc42a39d0323a61440eb12 in term 0.
I20260810 19:26:28.619421 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "0595e8ff3ebc42a39d0323a61440eb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de73a592a4e94614a914a184aceb2a87" is_pre_election: true
I20260810 19:26:28.619637 31863 leader_election.cc:302] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0595e8ff3ebc42a39d0323a61440eb12, ec40d07ba1ab434988bd05c03777fcce; no voters:
I20260810 19:26:28.619855 32252 raft_consensus.cc:2454] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0595e8ff3ebc42a39d0323a61440eb12 in term 0.
I20260810 19:26:28.620651 32639 raft_consensus.cc:2781] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:28.620779 32639 raft_consensus.cc:482] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:28.620870 32639 raft_consensus.cc:3037] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:28.623871 32639 raft_consensus.cc:504] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.624809 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "0595e8ff3ebc42a39d0323a61440eb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de73a592a4e94614a914a184aceb2a87"
I20260810 19:26:28.624882 32485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "0595e8ff3ebc42a39d0323a61440eb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40d07ba1ab434988bd05c03777fcce"
I20260810 19:26:28.624981 32252 raft_consensus.cc:3037] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:28.625144 32485 raft_consensus.cc:3037] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:28.626245 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.613046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:33861 (local address 127.19.68.3:42349)
0810 19:26:28.616138 (+ 3092us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.616143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.618026 (+ 1883us) server_negotiation.cc:407] Connection header received
0810 19:26:28.618616 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.618621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.618713 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.618763 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.619493 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.620493 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.623533 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.624903 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.625033 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.625173 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.625806 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.625942 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3001,"thread_start_us":5157,"threads_started":1}
I20260810 19:26:28.627065 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.612540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42349 (local address 127.19.68.2:33861)
0810 19:26:28.617923 (+ 5383us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.617943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.618466 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.618916 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.618919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.618939 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.619318 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.619331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.621245 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.621250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.623306 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.623321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.625360 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.625367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.625559 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.625602 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.626473 (+ 871us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.626629 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5086,"thread_start_us":4860,"threads_started":1}
I20260810 19:26:28.627831 32483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "de73a592a4e94614a914a184aceb2a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40d07ba1ab434988bd05c03777fcce" is_pre_election: true
I20260810 19:26:28.628295 32252 raft_consensus.cc:2454] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0595e8ff3ebc42a39d0323a61440eb12 in term 1.
I20260810 19:26:28.629187 31866 leader_election.cc:302] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0595e8ff3ebc42a39d0323a61440eb12, de73a592a4e94614a914a184aceb2a87; no voters:
I20260810 19:26:28.629536 32639 leader_election.cc:288] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [CANDIDATE]: Term 1 election: Requested vote from peers ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349), de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:28.629738 32639 raft_consensus.cc:2781] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:28.629957 32639 raft_consensus.cc:686] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 LEADER]: Becoming Leader. State: Replica: 0595e8ff3ebc42a39d0323a61440eb12, State: Running, Role: LEADER
I20260810 19:26:28.630038 32485 raft_consensus.cc:2454] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0595e8ff3ebc42a39d0323a61440eb12 in term 1.
I20260810 19:26:28.630493 32639 consensus_queue.cc:237] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } }
I20260810 19:26:28.635066 30391 catalog_manager.cc:5495] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "0595e8ff3ebc42a39d0323a61440eb12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } health_report { overall_health: HEALTHY } } }
I20260810 19:26:28.636992 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.613496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:46747 (local address 127.19.68.1:34767)
0810 19:26:28.613665 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.613673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.627893 (+ 14220us) server_negotiation.cc:407] Connection header received
0810 19:26:28.628141 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.628147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.628220 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.628286 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:28.628868 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.629941 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.632345 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.633686 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.633845 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.633947 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.636699 (+ 2752us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.636804 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 19:26:28.637108 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.612990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34767 (local address 127.19.68.2:46747)
0810 19:26:28.627799 (+ 14809us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:28.627818 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:28.627986 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:28.628487 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:28.628492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:28.628518 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:28.628697 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.628718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.630078 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.630084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.632143 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:28.632164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.636363 (+ 4199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.636368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:28.636547 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.636580 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:26:28.636678 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:26:28.636825 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14700,"thread_start_us":19585,"threads_started":1}
I20260810 19:26:28.638159 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c23633e3e546c68f43baa93efcd352" candidate_uuid: "de73a592a4e94614a914a184aceb2a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0595e8ff3ebc42a39d0323a61440eb12" is_pre_election: true
I20260810 19:26:28.639575 32135 leader_election.cc:302] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de73a592a4e94614a914a184aceb2a87; no voters: 0595e8ff3ebc42a39d0323a61440eb12, ec40d07ba1ab434988bd05c03777fcce
I20260810 19:26:28.639986 32638 raft_consensus.cc:2726] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 19:26:28.657281 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.651963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.19.68.1:34767)
0810 19:26:28.652129 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.652135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.652294 (+ 159us) server_negotiation.cc:407] Connection header received
0810 19:26:28.652485 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.652489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.652594 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.652727 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:28.653132 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.653886 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.655165 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.655606 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.655714 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.656181 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:28.656686 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:28.656811 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.656902 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.656982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:26:28.665294 32485 raft_consensus.cc:1261] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Refusing update from remote peer 0595e8ff3ebc42a39d0323a61440eb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:28.665378 32252 raft_consensus.cc:1261] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Refusing update from remote peer 0595e8ff3ebc42a39d0323a61440eb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:28.666278 32639 consensus_queue.cc:1035] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:28.666565 32651 consensus_queue.cc:1035] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:26:28.681059 32566 tablet.cc:2369] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:28.682526 32658 mvcc.cc:205] Tried to move back new op lower bound from 7317053393562816512 to 7317053393435009024. Current Snapshot: MvccSnapshot[applied={T|T < 7317053393562816512}]
I20260810 19:26:28.699793 30391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:28.700038 30391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:28.705653 30391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57286387165c4738b948e530dd4a0bef: Sending DeleteTablet for 3 replicas of tablet 89c23633e3e546c68f43baa93efcd352
I20260810 19:26:28.706960 32451 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c23633e3e546c68f43baa93efcd352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:28 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49938
I20260810 19:26:28.707996 32674 tablet_replica.cc:330] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: stopping tablet replica
I20260810 19:26:28.708297 32210 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c23633e3e546c68f43baa93efcd352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:28 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49106
I20260810 19:26:28.708420 32674 raft_consensus.cc:2229] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:28.708936 32674 raft_consensus.cc:2258] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:28.709510 32675 tablet_replica.cc:330] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: stopping tablet replica
I20260810 19:26:28.709965 32675 raft_consensus.cc:2229] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:28.710677 32675 raft_consensus.cc:2258] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:28.712260 31954 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c23633e3e546c68f43baa93efcd352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:28 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:43448
I20260810 19:26:28.717387 32674 ts_tablet_manager.cc:1905] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:28.719720 32675 ts_tablet_manager.cc:1905] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:28.724968 32674 ts_tablet_manager.cc:1918] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:28.725206 32674 log.cc:1196] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/89c23633e3e546c68f43baa93efcd352
I20260810 19:26:28.725857 32674 ts_tablet_manager.cc:1939] T 89c23633e3e546c68f43baa93efcd352 P ec40d07ba1ab434988bd05c03777fcce: Deleting consensus metadata
I20260810 19:26:28.726549 32681 tablet_replica.cc:330] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: stopping tablet replica
I20260810 19:26:28.727049 32681 raft_consensus.cc:2229] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:28.727932 32681 raft_consensus.cc:2258] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:28.727975 30363 catalog_manager.cc:4841] TS ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349): tablet 89c23633e3e546c68f43baa93efcd352 (table test-table [id=696675f089244c8e9a634e97cc1e9d22]) successfully deleted
I20260810 19:26:28.730592 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 30335
I20260810 19:26:28.731760 30438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:28.735224 32681 ts_tablet_manager.cc:1905] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:28.736732 32675 ts_tablet_manager.cc:1918] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:28.736933 32675 log.cc:1196] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/89c23633e3e546c68f43baa93efcd352
I20260810 19:26:28.737617 32675 ts_tablet_manager.cc:1939] T 89c23633e3e546c68f43baa93efcd352 P de73a592a4e94614a914a184aceb2a87: Deleting consensus metadata
I20260810 19:26:28.746780 32681 ts_tablet_manager.cc:1918] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:28.746970 32681 log.cc:1196] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/89c23633e3e546c68f43baa93efcd352
I20260810 19:26:28.747644 32681 ts_tablet_manager.cc:1939] T 89c23633e3e546c68f43baa93efcd352 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting consensus metadata
I20260810 19:26:28.748832 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.711508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.19.68.61:44511)
0810 19:26:28.711704 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.711711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.729394 (+ 17683us) server_negotiation.cc:407] Connection header received
0810 19:26:28.729775 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.729782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.729876 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.729961 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.732797 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.733989 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.736199 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.737358 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.737475 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.741546 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.741553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.741558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.741608 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.745333 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.746203 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.746209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.746213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.746371 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.746861 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.746866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.746869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.747240 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.747479 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.748237 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.748451 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:26:28.766978 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.729468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.19.68.60:44749)
0810 19:26:28.729662 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.729669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.750025 (+ 20356us) server_negotiation.cc:407] Connection header received
0810 19:26:28.750298 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.750304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.750376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.750429 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.751981 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.753293 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.755284 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.755984 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.756082 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.759993 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.760002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.760007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.760054 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.763454 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.764301 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.764307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.764311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.764475 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.765010 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.765016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.765020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.765460 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.765706 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.766289 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.766516 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:26:28.895706 30366 catalog_manager.cc:4841] TS de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205): tablet 89c23633e3e546c68f43baa93efcd352 (table test-table [id=696675f089244c8e9a634e97cc1e9d22]) successfully deleted
W20260810 19:26:28.895979 30366 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.19.68.2:33205, user_credentials={real_user=slave}} blocked reactor thread for 156378us
I20260810 19:26:28.896409 30366 catalog_manager.cc:4841] TS 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767): tablet 89c23633e3e546c68f43baa93efcd352 (table test-table [id=696675f089244c8e9a634e97cc1e9d22]) successfully deleted
I20260810 19:26:28.896721 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 30335
I20260810 19:26:28.947592 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.931121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.19.68.61:44511)
0810 19:26:28.931359 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.931365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.931782 (+ 417us) server_negotiation.cc:407] Connection header received
0810 19:26:28.932023 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.932031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.932126 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.932262 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.934336 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.935530 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.937193 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.937684 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.937906 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.941867 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.941873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.941878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.941921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.944568 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.945210 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.945214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.945217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.945347 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.945684 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.945688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.945690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.946088 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.946341 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.946883 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.947122 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:26:28.948455 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:28.931536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.19.68.60:44749)
0810 19:26:28.931769 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:28.931774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:28.932695 (+ 921us) server_negotiation.cc:407] Connection header received
0810 19:26:28.932989 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:28.932995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:28.933103 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:28.933214 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:28.936370 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.937532 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.939189 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:28.939661 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:28.939775 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:28.942222 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:28.942230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:28.942235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:28.942280 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:28.945693 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.946439 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.946444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.946448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.946625 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:28.946996 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:28.947001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:28.947005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:28.947322 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:28.947557 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:28.947962 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 19:26:28.948150 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:26:29.690050 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:29.689024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:38541)
0810 19:26:29.689624 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:29.689806 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
W20260810 19:26:29.690495 32562 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:42309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:29.696084 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:29.694898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:50953)
0810 19:26:29.695718 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:29.695879 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":590,"threads_started":1}
W20260810 19:26:29.696667 32326 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:42309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:29.697067 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:29.696278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:39709)
0810 19:26:29.696771 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:29.696909 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
W20260810 19:26:29.697458 32078 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:42309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:30.207680 32696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57286387165c4738b948e530dd4a0bef)
I20260810 19:26:30.207654 32695 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57286387165c4738b948e530dd4a0bef)
I20260810 19:26:30.207978 32696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } }
I20260810 19:26:30.207988 32695 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } }
I20260810 19:26:30.209713 32696 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309), f39095eeb1164695aad63abd8ff8405d (127.19.68.60:44749)
I20260810 19:26:30.209713 32695 leader_election.cc:288] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309), 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61:44511)
I20260810 19:26:30.214022 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.209977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40268)
0810 19:26:30.213496 (+ 3519us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.213655 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3393,"thread_start_us":3265,"threads_started":1}
I20260810 19:26:30.214733 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.209964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40254)
0810 19:26:30.214267 (+ 4303us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.214480 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4078,"thread_start_us":4130,"threads_started":1}
W20260810 19:26:30.215330 31404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
W20260810 19:26:30.216077 31713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:30.223758 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.213811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:44511 (local address 127.0.0.1:52076)
0810 19:26:30.214227 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.214244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.214388 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.216462 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.216467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.216499 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.216762 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.216774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.218759 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.218766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.221029 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.221049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.222908 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.222915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.223123 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.223161 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.223237 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.223389 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":810,"threads_started":1}
I20260810 19:26:30.224725 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.224309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40272)
0810 19:26:30.224454 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.224541 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:26:30.225039 31713 leader_election.cc:334] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309): Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:30.225235 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.213895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.19.68.61:44511)
0810 19:26:30.216031 (+ 2136us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.216036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.216057 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:26:30.216121 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.216126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.216196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.216907 (+ 711us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.217180 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.218536 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.221297 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.222690 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.224451 (+ 1761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.224577 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.224650 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.224794 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2019,"thread_start_us":1782,"threads_started":1}
I20260810 19:26:30.225929 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39095eeb1164695aad63abd8ff8405d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8ac017532d5f4f189775bbefda2e28ba" is_pre_election: true
I20260810 19:26:30.226408 31442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39095eeb1164695aad63abd8ff8405d in term 1.
I20260810 19:26:30.227020 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.225257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40280)
0810 19:26:30.226456 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.226666 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":689,"threads_started":1}
I20260810 19:26:30.227000 31713 leader_election.cc:302] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ac017532d5f4f189775bbefda2e28ba, f39095eeb1164695aad63abd8ff8405d; no voters: 57286387165c4738b948e530dd4a0bef
W20260810 19:26:30.227303 31404 leader_election.cc:334] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309): Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:30.227375 32695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 19:26:30.227499 32695 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57286387165c4738b948e530dd4a0bef)
I20260810 19:26:30.227602 32695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:26:30.230000 32695 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } }
I20260810 19:26:30.230276 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.214900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44749 (local address 127.0.0.1:59266)
0810 19:26:30.215121 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.215141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.215287 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.223977 (+ 8690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.223982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.224035 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.224413 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.224428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.225904 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.225915 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.228033 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.228056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.229458 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.229465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.229653 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.229680 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.229769 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.229923 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":60}
I20260810 19:26:30.230384 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.214843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.19.68.60:44749)
0810 19:26:30.223131 (+ 8288us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.223136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.223153 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:26:30.223216 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.223220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.223308 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.223401 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.224609 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.225693 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.228235 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.229246 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.229393 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.229516 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.229824 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.229971 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8152,"thread_start_us":7991,"threads_started":1}
I20260810 19:26:30.230729 32695 leader_election.cc:288] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 election: Requested vote from peers 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309), 8ac017532d5f4f189775bbefda2e28ba (127.19.68.61:44511)
I20260810 19:26:30.231127 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.230799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40292)
0810 19:26:30.230926 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.231011 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 19:26:30.230993 31754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ac017532d5f4f189775bbefda2e28ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f39095eeb1164695aad63abd8ff8405d" is_pre_election: true
I20260810 19:26:30.231438 31754 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac017532d5f4f189775bbefda2e28ba in current term 2: Already voted for candidate f39095eeb1164695aad63abd8ff8405d in this term.
I20260810 19:26:30.231765 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39095eeb1164695aad63abd8ff8405d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8ac017532d5f4f189775bbefda2e28ba"
I20260810 19:26:30.231993 31442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:26:30.232120 31404 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac017532d5f4f189775bbefda2e28ba; no voters: 57286387165c4738b948e530dd4a0bef, f39095eeb1164695aad63abd8ff8405d
I20260810 19:26:30.234324 31442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39095eeb1164695aad63abd8ff8405d in term 2.
I20260810 19:26:30.234519 32696 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:26:30.234858 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.231892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40298)
0810 19:26:30.234630 (+ 2738us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.234715 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2635}
I20260810 19:26:30.234854 31713 leader_election.cc:302] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac017532d5f4f189775bbefda2e28ba, f39095eeb1164695aad63abd8ff8405d; no voters:
W20260810 19:26:30.235172 31713 leader_election.cc:334] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309): Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
I20260810 19:26:30.235203 32695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:26:30.235663 32695 raft_consensus.cc:686] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [term 2 LEADER]: Becoming Leader. State: Replica: f39095eeb1164695aad63abd8ff8405d, State: Running, Role: LEADER
I20260810 19:26:30.236085 32695 consensus_queue.cc:237] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } }
I20260810 19:26:30.237790 32715 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39095eeb1164695aad63abd8ff8405d. Latest consensus state: current_term: 2 leader_uuid: "f39095eeb1164695aad63abd8ff8405d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:30.238102 32715 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:30.238750 32720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:30.242707 32720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:30.243498 32720 catalog_manager.cc:1259] Loaded cluster ID: a0a171afd5874fdaafd1137337b2a4a2
I20260810 19:26:30.243574 32720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:30.245471 32720 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:30.246292 32720 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Loaded TSK: 0
I20260810 19:26:30.247162 32720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:26:30.303275 31739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:30.310436 31739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:30.314469 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.314074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40330)
0810 19:26:30.314226 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.314315 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 19:26:30.314797 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [term 2 FOLLOWER]: Refusing update from remote peer f39095eeb1164695aad63abd8ff8405d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 19:26:30.315447 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.315081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40344)
0810 19:26:30.315237 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.315306 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:26:30.315589 31713 consensus_peers.cc:487] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d -> Peer 57286387165c4738b948e530dd4a0bef (127.19.68.62:42309): Couldn't send request to peer 57286387165c4738b948e530dd4a0bef. Status: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 19:26:30.315610 32695 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 19:26:30.318066 32696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39095eeb1164695aad63abd8ff8405d. Latest consensus state: current_term: 2 leader_uuid: "f39095eeb1164695aad63abd8ff8405d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:30.318349 32696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:30.319931 32695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f39095eeb1164695aad63abd8ff8405d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:30.320310 32695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:30.320673 32696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f39095eeb1164695aad63abd8ff8405d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57286387165c4738b948e530dd4a0bef" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 42309 } } peers { permanent_uuid: "8ac017532d5f4f189775bbefda2e28ba" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 44511 } attrs { promote: false } } peers { permanent_uuid: "f39095eeb1164695aad63abd8ff8405d" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44749 } attrs { promote: false } } }
I20260810 19:26:30.320951 32696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac017532d5f4f189775bbefda2e28ba [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:26:30.334792 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.328786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34767 (local address 127.0.0.1:43454)
0810 19:26:30.328960 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.328978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.329110 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.329778 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.329783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.329808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.330144 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.330154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.331328 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.331333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.332770 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.332782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.334057 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.334064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.334232 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.334257 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.334339 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.334475 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 19:26:30.334784 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.328905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.19.68.1:34767)
0810 19:26:30.329510 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.329515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.329529 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:26:30.329585 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.329589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.329653 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.329737 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.330295 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.331176 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.332946 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.333837 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.334008 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.334109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.334377 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.334511 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":402,"threads_started":1}
I20260810 19:26:30.341933 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.328266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42349 (local address 127.0.0.1:49954)
0810 19:26:30.328413 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.328431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.328532 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.337662 (+ 9130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.337667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.337691 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.337828 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.337836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.338805 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.338810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.340026 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.340040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.341265 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.341271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.341437 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.341462 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.341538 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.341659 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 19:26:30.341989 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.328422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.19.68.3:42349)
0810 19:26:30.337403 (+ 8981us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.337408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.337423 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:26:30.337480 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.337483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.337537 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.337623 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.337971 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.338676 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.340225 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.341129 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.341251 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.341349 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.341584 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.341725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8877,"thread_start_us":8799,"threads_started":1}
I20260810 19:26:30.343118 32451 tablet_service.cc:1467] Processing CreateTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (DEFAULT_TABLE table=test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:30.343616 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:30.346477 31954 tablet_service.cc:1467] Processing CreateTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (DEFAULT_TABLE table=test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:30.346871 31954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:30.349678 32734 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Bootstrap starting.
I20260810 19:26:30.351061 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.329604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33205 (local address 127.0.0.1:49116)
0810 19:26:30.335003 (+ 5399us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.335018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.335102 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.345457 (+ 10355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.345462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.345489 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.345671 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.345680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.347069 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.347074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.348889 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.348915 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.350360 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.350366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.350539 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.350562 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.350643 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.350775 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5296,"thread_start_us":16098,"threads_started":1}
I20260810 19:26:30.351101 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.329680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.19.68.2:33205)
0810 19:26:30.345100 (+ 15420us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.345108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.345131 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:26:30.345210 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.345216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.345306 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.345420 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.345860 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.346919 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.349127 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.350210 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.350323 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.350423 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.350682 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.350818 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15255,"thread_start_us":16177,"threads_started":1}
I20260810 19:26:30.351652 32734 tablet_bootstrap.cc:654] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:30.351758 32210 tablet_service.cc:1467] Processing CreateTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (DEFAULT_TABLE table=test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:30.352269 32210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:30.357005 32738 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Bootstrap starting.
I20260810 19:26:30.358937 32737 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Bootstrap starting.
I20260810 19:26:30.358968 32738 tablet_bootstrap.cc:654] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:30.361109 32737 tablet_bootstrap.cc:654] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:30.361624 32734 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: No bootstrap required, opened a new log
I20260810 19:26:30.361719 32738 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: No bootstrap required, opened a new log
I20260810 19:26:30.361759 32734 ts_tablet_manager.cc:1397] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260810 19:26:30.361907 32738 ts_tablet_manager.cc:1397] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:26:30.362511 32734 raft_consensus.cc:348] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.362699 32734 raft_consensus.cc:374] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:30.362784 32734 raft_consensus.cc:729] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40d07ba1ab434988bd05c03777fcce, State: Initialized, Role: FOLLOWER
I20260810 19:26:30.362789 32738 raft_consensus.cc:348] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.363042 32738 raft_consensus.cc:374] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:30.363154 32738 raft_consensus.cc:729] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de73a592a4e94614a914a184aceb2a87, State: Initialized, Role: FOLLOWER
I20260810 19:26:30.363047 32734 consensus_queue.cc:260] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.363413 32738 consensus_queue.cc:260] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.363842 32734 ts_tablet_manager.cc:1428] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:26:30.369450 32737 tablet_bootstrap.cc:492] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: No bootstrap required, opened a new log
I20260810 19:26:30.369606 32737 ts_tablet_manager.cc:1397] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260810 19:26:30.370460 32737 raft_consensus.cc:348] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.370687 32737 raft_consensus.cc:374] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:30.370791 32737 raft_consensus.cc:729] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0595e8ff3ebc42a39d0323a61440eb12, State: Initialized, Role: FOLLOWER
I20260810 19:26:30.371079 32737 consensus_queue.cc:260] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.371304 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.364393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:49949)
0810 19:26:30.370873 (+ 6480us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.371022 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6334,"thread_start_us":6214,"threads_started":1}
I20260810 19:26:30.374694 32561 heartbeater.cc:503] Master 127.19.68.60:44749 was elected leader, sending a full tablet report...
I20260810 19:26:30.375144 32738 ts_tablet_manager.cc:1428] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20260810 19:26:30.376273 32737 ts_tablet_manager.cc:1428] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260810 19:26:30.376782 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.372178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:43927)
0810 19:26:30.375272 (+ 3094us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.376526 (+ 1254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2951,"thread_start_us":2815,"threads_started":1}
I20260810 19:26:30.377378 32324 heartbeater.cc:503] Master 127.19.68.60:44749 was elected leader, sending a full tablet report...
I20260810 19:26:30.378579 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.376114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:60915)
0810 19:26:30.378133 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.378286 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":364,"threads_started":1}
I20260810 19:26:30.378721 32073 heartbeater.cc:503] Master 127.19.68.60:44749 was elected leader, sending a full tablet report...
I20260810 19:26:30.378902 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.378482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:59329)
0810 19:26:30.378678 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.378762 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 19:26:30.379781 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.375428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:58879)
0810 19:26:30.379195 (+ 3767us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.379300 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3675}
I20260810 19:26:30.380106 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.379626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:54501)
0810 19:26:30.379838 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.380000 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 19:26:30.388717 32743 raft_consensus.cc:482] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:30.388936 32743 raft_consensus.cc:504] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.390690 32743 leader_election.cc:288] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767), de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:30.397372 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.390880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:34767 (local address 127.19.68.3:57135)
0810 19:26:30.391028 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.391049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.391239 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.391772 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.391778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.391814 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.392104 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.392119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.393396 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.393403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.395194 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.395217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.396525 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.396532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.396726 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.396767 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.396879 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.397027 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 19:26:30.397732 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.391307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:33205 (local address 127.19.68.3:58721)
0810 19:26:30.391950 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.391969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:30.392092 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:30.392441 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:30.392446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:30.392472 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:30.392766 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.392777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.394047 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.394050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.395578 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:30.395593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.396898 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.396904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:30.397072 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.397123 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 19:26:30.397209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:26:30.397330 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":360,"threads_started":1}
I20260810 19:26:30.398061 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.391402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:58721 (local address 127.19.68.2:33205)
0810 19:26:30.391665 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.391672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.392059 (+ 387us) server_negotiation.cc:407] Connection header received
0810 19:26:30.392211 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.392217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.392298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.392379 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.392914 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.393915 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.395756 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.396693 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.396842 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.396960 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.397271 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.397406 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:26:30.398247 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.390963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:57135 (local address 127.19.68.1:34767)
0810 19:26:30.391169 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.391175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.391197 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:26:30.391458 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.391465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.391558 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.391661 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:30.392259 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.393233 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.395380 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.396320 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.396416 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.396491 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.396895 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.397966 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:30.398711 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb5cbcfbdc94212bf9e3fd8182a8be8" candidate_uuid: "ec40d07ba1ab434988bd05c03777fcce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de73a592a4e94614a914a184aceb2a87" is_pre_election: true
I20260810 19:26:30.398761 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb5cbcfbdc94212bf9e3fd8182a8be8" candidate_uuid: "ec40d07ba1ab434988bd05c03777fcce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0595e8ff3ebc42a39d0323a61440eb12" is_pre_election: true
I20260810 19:26:30.398964 32252 raft_consensus.cc:2454] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec40d07ba1ab434988bd05c03777fcce in term 0.
I20260810 19:26:30.399015 31980 raft_consensus.cc:2454] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec40d07ba1ab434988bd05c03777fcce in term 0.
I20260810 19:26:30.399507 32387 leader_election.cc:302] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0595e8ff3ebc42a39d0323a61440eb12, ec40d07ba1ab434988bd05c03777fcce; no voters:
I20260810 19:26:30.399865 32743 raft_consensus.cc:2781] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:30.399986 32743 raft_consensus.cc:482] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:30.400087 32743 raft_consensus.cc:3037] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:30.402196 32743 raft_consensus.cc:504] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.402714 32743 leader_election.cc:288] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [CANDIDATE]: Term 1 election: Requested vote from peers 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767), de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205)
I20260810 19:26:30.403208 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb5cbcfbdc94212bf9e3fd8182a8be8" candidate_uuid: "ec40d07ba1ab434988bd05c03777fcce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0595e8ff3ebc42a39d0323a61440eb12"
I20260810 19:26:30.403290 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb5cbcfbdc94212bf9e3fd8182a8be8" candidate_uuid: "ec40d07ba1ab434988bd05c03777fcce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de73a592a4e94614a914a184aceb2a87"
I20260810 19:26:30.403445 31980 raft_consensus.cc:3037] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:30.403498 32252 raft_consensus.cc:3037] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:30.405844 32252 raft_consensus.cc:2454] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec40d07ba1ab434988bd05c03777fcce in term 1.
I20260810 19:26:30.406076 31980 raft_consensus.cc:2454] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec40d07ba1ab434988bd05c03777fcce in term 1.
I20260810 19:26:30.406206 32387 leader_election.cc:302] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de73a592a4e94614a914a184aceb2a87, ec40d07ba1ab434988bd05c03777fcce; no voters:
I20260810 19:26:30.406539 32743 raft_consensus.cc:2781] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:30.407008 32743 raft_consensus.cc:686] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 1 LEADER]: Becoming Leader. State: Replica: ec40d07ba1ab434988bd05c03777fcce, State: Running, Role: LEADER
I20260810 19:26:30.407439 32743 consensus_queue.cc:237] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } }
I20260810 19:26:30.407985 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.407617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:59675)
0810 19:26:30.407775 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.407861 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:26:30.411175 31739 catalog_manager.cc:5495] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce reported cstate change: term changed from 0 to 1, leader changed from <none> to ec40d07ba1ab434988bd05c03777fcce (127.19.68.3). New cstate: current_term: 1 leader_uuid: "ec40d07ba1ab434988bd05c03777fcce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40d07ba1ab434988bd05c03777fcce" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:30.445413 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.440357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.19.68.3:42349)
0810 19:26:30.440553 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.440559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.440580 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:26:30.440820 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.440824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.440913 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.441044 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:30.441427 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.442104 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.443327 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.443735 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.443856 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.444293 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:30.444813 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:30.444951 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.445053 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.445158 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 19:26:30.449134 32252 raft_consensus.cc:1261] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Refusing update from remote peer ec40d07ba1ab434988bd05c03777fcce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:30.449541 31980 raft_consensus.cc:1261] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Refusing update from remote peer ec40d07ba1ab434988bd05c03777fcce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:30.449872 32755 consensus_queue.cc:1035] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [LEADER]: Connected to new peer: Peer: permanent_uuid: "de73a592a4e94614a914a184aceb2a87" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:30.450218 32743 consensus_queue.cc:1035] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0595e8ff3ebc42a39d0323a61440eb12" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:30.452700 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.452134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:57067)
0810 19:26:30.452382 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.452496 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260810 19:26:30.453328 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.451841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:45151)
0810 19:26:30.453030 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.453151 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1066}
I20260810 19:26:30.459311 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.458886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.1:58135)
0810 19:26:30.459065 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.459162 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 19:26:30.469632 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.465669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:35961)
0810 19:26:30.469385 (+ 3716us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.469479 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3611}
I20260810 19:26:30.473274 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.472520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.2:55779)
0810 19:26:30.472915 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.473019 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
I20260810 19:26:30.473229 32765 mvcc.cc:205] Tried to move back new op lower bound from 7317053400873365504 to 7317053400713269248. Current Snapshot: MvccSnapshot[applied={T|T < 7317053400873365504}]
I20260810 19:26:30.476680 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.473982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.19.68.3:51263)
0810 19:26:30.476379 (+ 2397us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.476476 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2283}
I20260810 19:26:30.486971 31738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:30.487265 31738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:30.492929 31738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f39095eeb1164695aad63abd8ff8405d: Sending DeleteTablet for 3 replicas of tablet dcb5cbcfbdc94212bf9e3fd8182a8be8
I20260810 19:26:30.493901 32451 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:30 UTC) from {username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:49954
I20260810 19:26:30.494494 32210 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:30 UTC) from {username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:30.494506 31954 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:30 UTC) from {username='oryx', principal='oryx/127.19.68.60@KRBTEST.COM'} at 127.0.0.1:43454
I20260810 19:26:30.495757 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 31721
I20260810 19:26:30.498667 316 tablet_replica.cc:330] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: stopping tablet replica
I20260810 19:26:30.498878 316 raft_consensus.cc:2229] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:30.499493 316 raft_consensus.cc:2258] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:30.500692 316 ts_tablet_manager.cc:1905] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:30.505511 319 tablet_replica.cc:330] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: stopping tablet replica
I20260810 19:26:30.505692 319 raft_consensus.cc:2229] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:30.505977 319 raft_consensus.cc:2258] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:30.506539 321 tablet_replica.cc:330] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: stopping tablet replica
I20260810 19:26:30.506716 321 raft_consensus.cc:2229] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:30.506979 321 raft_consensus.cc:2258] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:30.508242 321 ts_tablet_manager.cc:1905] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:30.508349 319 ts_tablet_manager.cc:1905] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:30.512914 319 ts_tablet_manager.cc:1918] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:30.513041 319 log.cc:1196] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/dcb5cbcfbdc94212bf9e3fd8182a8be8
I20260810 19:26:30.513547 319 ts_tablet_manager.cc:1939] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P de73a592a4e94614a914a184aceb2a87: Deleting consensus metadata
I20260810 19:26:30.514709 31719 catalog_manager.cc:4841] TS de73a592a4e94614a914a184aceb2a87 (127.19.68.2:33205): tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (table test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]) successfully deleted
I20260810 19:26:30.515081 321 ts_tablet_manager.cc:1918] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:30.515234 321 log.cc:1196] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/dcb5cbcfbdc94212bf9e3fd8182a8be8
I20260810 19:26:30.515745 321 ts_tablet_manager.cc:1939] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P 0595e8ff3ebc42a39d0323a61440eb12: Deleting consensus metadata
I20260810 19:26:30.516885 31719 catalog_manager.cc:4841] TS 0595e8ff3ebc42a39d0323a61440eb12 (127.19.68.1:34767): tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (table test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]) successfully deleted
I20260810 19:26:30.516937 316 ts_tablet_manager.cc:1918] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:30.517069 316 log.cc:1196] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/dcb5cbcfbdc94212bf9e3fd8182a8be8
I20260810 19:26:30.517503 316 ts_tablet_manager.cc:1939] T dcb5cbcfbdc94212bf9e3fd8182a8be8 P ec40d07ba1ab434988bd05c03777fcce: Deleting consensus metadata
I20260810 19:26:30.518594 31717 catalog_manager.cc:4841] TS ec40d07ba1ab434988bd05c03777fcce (127.19.68.3:42349): tablet dcb5cbcfbdc94212bf9e3fd8182a8be8 (table test-table [id=1110a8b6798d4b3c98bcefd31c9c71e9]) successfully deleted
I20260810 19:26:30.530020 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.497260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.19.68.1:34767)
0810 19:26:30.497447 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.497454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.507607 (+ 10153us) server_negotiation.cc:407] Connection header received
0810 19:26:30.507777 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.507782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.507873 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.507973 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:30.517846 (+ 9873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.518846 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.520588 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.520965 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.521058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.524535 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:30.524549 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:30.524564 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:30.524594 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:30.527319 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.528085 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.528092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.528094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.528207 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.528611 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.528615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.528616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.528963 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:30.529221 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.529573 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.529738 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:26:30.530709 32050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:30.651253 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 31721
I20260810 19:26:30.682657 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 32106
I20260810 19:26:30.697830 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.683429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.19.68.2:33205)
0810 19:26:30.683622 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.683626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.683792 (+ 166us) server_negotiation.cc:407] Connection header received
0810 19:26:30.683985 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.683989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.684053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.684132 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:30.685958 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.686677 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.688446 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.688837 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.688932 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.692192 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:30.692207 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:30.692222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:30.692261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:30.695172 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.695892 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.695906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.695909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.696024 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.696357 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.696359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.696361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.696704 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:30.696943 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.697337 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.697506 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:26:30.698478 32315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:30.756228 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.755690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40358)
0810 19:26:30.755924 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.756045 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 19:26:30.757743 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.757268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:42309 (local address 127.0.0.1:40368)
0810 19:26:30.757483 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:30.757583 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260810 19:26:30.806277 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 32106
I20260810 19:26:30.831517 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 32352
I20260810 19:26:30.845638 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:30.832372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.19.68.3:42349)
0810 19:26:30.832593 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:30.832600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:30.832619 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:26:30.832802 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:30.832806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:30.832893 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:30.832985 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:30.834346 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.835141 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.836436 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:30.836854 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:30.836964 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:30.839525 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:30.839539 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:30.839553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:30.839594 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:30.842827 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.843481 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.843493 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.843495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.843645 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:30.844006 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:30.844010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:30.844013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:30.844387 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:30.844709 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:30.844984 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 19:26:30.845306 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:26:30.846407 32554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:30.978884 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 32352
I20260810 19:26:31.008953 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 31198
I20260810 19:26:31.010097 31478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:31.127667 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 31198
I20260810 19:26:31.156057 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 31363
I20260810 19:26:31.157152 31792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:31.278311 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 31363
2026-08-10T19:26:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9715 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:31 dist-test-slave-2j7r krb5kdc[341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:31 dist-test-slave-2j7r krb5kdc[341](info): set up 2 sockets
Aug 10 19:26:31 dist-test-slave-2j7r krb5kdc[341](info): commencing operation
krb5kdc: starting...
W20260810 19:26:32.570607 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:32 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab.
Entry for principal oryx/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab.
I20260810 19:26:32.830080 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44569
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab
--principal=oryx/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:33.160290 367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:33.160562 367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:33.160681 367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:33.167761 367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:33.167855 367 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:26:33.167913 367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:33.168001 367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:33.168066 367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:33.168152 367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:33.177760 367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:44569
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.62.keytab
--principal=oryx/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:33.178983 367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:33.180315 367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:33.192008 381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:33.192045 382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:33.191977 384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:33.193352 367 server_base.cc:1034] running on GCE node
I20260810 19:26:33.194125 367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:33.195396 367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:33.196655 367 hybrid_clock.cc:648] HybridClock initialized: now 1786389993196572 us; error 33 us; skew 500 ppm
I20260810 19:26:33.197036 367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:33 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389993, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:33.200341 367 init.cc:377] Logged in from keytab as oryx/127.19.68.62@KRBTEST.COM (short username oryx)
I20260810 19:26:33.201982 367 webserver.cc:460] Webserver started at http://127.19.68.62:45995/ using document root <none> and password file <none>
I20260810 19:26:33.202749 367 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:33.202868 367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:33.203241 367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:33.206579 367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "28908ffa30b9442f8da979a17a30406e"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "4e02c59b476d2206c9c389d898d3a005"
server_key_iv: "1c27bdebf3c79c3cfe09a10c2e0faa94"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.207455 367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "28908ffa30b9442f8da979a17a30406e"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "4e02c59b476d2206c9c389d898d3a005"
server_key_iv: "1c27bdebf3c79c3cfe09a10c2e0faa94"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.213050 367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:26:33.216683 402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:33.218101 367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:33.218293 367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "28908ffa30b9442f8da979a17a30406e"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "4e02c59b476d2206c9c389d898d3a005"
server_key_iv: "1c27bdebf3c79c3cfe09a10c2e0faa94"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.218499 367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:33.232738 367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:33.236778 367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:33.237030 367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:33.257558 367 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:44569
I20260810 19:26:33.257579 506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:44569 every 8 connection(s)
I20260810 19:26:33.258872 367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:33.262463 507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:33.265172 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 367
I20260810 19:26:33.265364 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:33.266026 19728 external_mini_cluster.cc:1428] Setting key 6428efb16d47082ce3e9a3f2b2f98a2f
I20260810 19:26:33.275010 507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Bootstrap starting.
I20260810 19:26:33.278363 507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:33.279304 507 log.cc:824] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:26:33 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:33.282187 507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: No bootstrap required, opened a new log
I20260810 19:26:33.289714 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:33.267920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.19.68.62:44569)
0810 19:26:33.268745 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:33.268760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:33.268810 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:26:33.270918 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:33.270950 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:33.271431 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:33.271859 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:33.273179 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.275082 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.276338 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.277381 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.277557 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:33.281509 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:33.281520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:33.281536 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:33.281580 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:33.285431 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.286092 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.286100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.286107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.286319 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.286638 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.286643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.286645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.287323 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:33.287626 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:33.287953 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 19:26:33.288375 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":353,"threads_started":1}
I20260810 19:26:33.291905 507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } }
I20260810 19:26:33.292232 507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:33.292338 507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28908ffa30b9442f8da979a17a30406e, State: Initialized, Role: FOLLOWER
I20260810 19:26:33.292865 507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } }
I20260810 19:26:33.293071 507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:33.293243 507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:33.293360 507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:33.295336 507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } }
I20260810 19:26:33.295774 507 leader_election.cc:302] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28908ffa30b9442f8da979a17a30406e; no voters:
I20260810 19:26:33.296496 507 leader_election.cc:288] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:33.296610 515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:33.297358 515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [term 1 LEADER]: Becoming Leader. State: Replica: 28908ffa30b9442f8da979a17a30406e, State: Running, Role: LEADER
I20260810 19:26:33.297912 515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } }
I20260810 19:26:33.298454 507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:33.300686 522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28908ffa30b9442f8da979a17a30406e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } } }
I20260810 19:26:33.301180 522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:33.300659 524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28908ffa30b9442f8da979a17a30406e. Latest consensus state: current_term: 1 leader_uuid: "28908ffa30b9442f8da979a17a30406e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28908ffa30b9442f8da979a17a30406e" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44569 } } }
I20260810 19:26:33.301728 524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:33.302845 540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:33.308178 540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:33.317243 540 catalog_manager.cc:1347] Generated new cluster ID: 40831f74ce58473d957c969c6478596a
I20260810 19:26:33.317333 540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:33.328862 540 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:33.330094 540 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:33.340557 540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Generated new TSK 0
I20260810 19:26:33.341326 540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
I20260810 19:26:33.412112 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44569
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:33.754031 555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:33.754292 555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:33.754396 555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:33.761514 555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:33.761611 555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:33.761699 555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:26:33.761777 555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:33.771193 555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:33.772326 555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:33.773669 555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:33.786271 571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:33.786221 568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:33.786448 567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:33.787587 555 server_base.cc:1034] running on GCE node
I20260810 19:26:33.788142 555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:33.789214 555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:33.790462 555 hybrid_clock.cc:648] HybridClock initialized: now 1786389993790395 us; error 33 us; skew 500 ppm
I20260810 19:26:33.790755 555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:33 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389993, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:33.793910 555 init.cc:377] Logged in from keytab as oryx/127.19.68.1@KRBTEST.COM (short username oryx)
I20260810 19:26:33.795445 555 webserver.cc:460] Webserver started at http://127.19.68.1:43575/ using document root <none> and password file <none>
I20260810 19:26:33.796146 555 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:33.796271 555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:33.796617 555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:33.799860 555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "328810c709c84794889c6b3d12221d27"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "bce15508326d2467d409bf18cb16a047"
server_key_iv: "292b5ee55aa7e2c0b4dc94dc36669340"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.800697 555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "328810c709c84794889c6b3d12221d27"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "bce15508326d2467d409bf18cb16a047"
server_key_iv: "292b5ee55aa7e2c0b4dc94dc36669340"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.805958 555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 19:26:33.809345 583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:33.810683 555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:26:33.810844 555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "328810c709c84794889c6b3d12221d27"
format_stamp: "Formatted at 2026-08-10 19:26:33 on dist-test-slave-2j7r"
server_key: "bce15508326d2467d409bf18cb16a047"
server_key_iv: "292b5ee55aa7e2c0b4dc94dc36669340"
server_key_version: "encryptionkey@0"
I20260810 19:26:33.811038 555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:33.824851 555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:33.828737 555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:33.828994 555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:33.831661 555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:33.831822 555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 19:26:33.831983 555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:33.832048 555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:26:33 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389993, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:33.853802 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:33.831960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51125 (local address 127.19.68.62:44569)
0810 19:26:33.832482 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:33.832489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:33.833545 (+ 1056us) server_negotiation.cc:407] Connection header received
0810 19:26:33.835529 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:33.835535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:33.835627 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:33.835722 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:33.837755 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.838944 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.841332 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.841969 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.842112 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:33.846795 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:33.846803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:33.846809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:33.846859 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:33.850058 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.851169 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.851175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.851179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.851330 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.851828 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.851831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.851833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.852111 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:33.852225 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:33.853162 (+ 937us) server_negotiation.cc:299] Negotiation successful
0810 19:26:33.853420 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":243,"threads_started":1}
I20260810 19:26:33.855568 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:33.832478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.1:51125)
0810 19:26:33.833360 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:33.833427 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:33.835247 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:33.835978 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:33.835987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:33.836704 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:33.837542 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:33.837571 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.839135 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.839140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:33.841167 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:33.841185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.842424 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.842433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:33.842713 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:33.843572 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:33.843612 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:33.846522 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:33.850241 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:33.850251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:33.850284 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:33.851017 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:33.851447 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:33.851453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:33.851457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:33.851730 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:33.852230 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:33.852240 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:33.852710 (+ 470us) client_negotiation.cc:769] Sending connection context
0810 19:26:33.853041 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 19:26:33.853756 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":347,"threads_started":1}
I20260810 19:26:33.873925 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:33.861173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51793 (local address 127.19.68.62:44569)
0810 19:26:33.861343 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:33.861348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:33.861947 (+ 599us) server_negotiation.cc:407] Connection header received
0810 19:26:33.862132 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:33.862137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:33.862215 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:33.862310 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:33.863509 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.864384 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.866338 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.866751 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.866857 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:33.868544 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:33.868548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:33.868551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:33.868580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:33.871369 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.872044 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.872047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.872049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.872167 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:33.872510 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:33.872514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:33.872518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:33.872781 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:33.872920 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:33.873374 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 19:26:33.873596 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:33.874065 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:33.861137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.1:51793)
0810 19:26:33.861828 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:33.861848 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:33.862005 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:33.862424 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:33.862430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:33.863038 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:33.863370 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:33.863381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.864514 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.864519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:33.866206 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:33.866219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:33.866934 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:33.866941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:33.867094 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:33.867677 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:33.867693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:33.868372 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:33.871541 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:33.871545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:33.871548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:33.871918 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:33.872274 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:33.872279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:33.872283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:33.872410 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:33.872892 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:33.872905 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:33.873079 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 19:26:33.873258 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 19:26:33.873594 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":356,"threads_started":1}
I20260810 19:26:33.883445 555 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:39617
I20260810 19:26:33.883464 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:39617 every 8 connection(s)
I20260810 19:26:33.884665 555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:33.886768 804 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44569
I20260810 19:26:33.886968 804 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:33.887596 804 heartbeater.cc:511] Master 127.19.68.62:44569 requested a full tablet report, sending...
I20260810 19:26:33.889524 429 ts_manager.cc:194] Registered new tserver with Master: 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617)
I20260810 19:26:33.890925 429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:51125
I20260810 19:26:33.892149 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 555
I20260810 19:26:33.892388 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:33.893018 19728 external_mini_cluster.cc:1428] Setting key 96cb7f2218470e4dfe239532e13c8a6d
WARNING: no policy specified for oryx/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
I20260810 19:26:33.950317 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44569
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:34.282858 809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:34.283142 809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:34.283277 809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:34.290526 809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:34.290632 809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:34.290751 809 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:26:34.290854 809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:34.300745 809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:34.302016 809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:34.303393 809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:34.316064 821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:34.316098 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:34.315994 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:34.317229 809 server_base.cc:1034] running on GCE node
I20260810 19:26:34.317791 809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:34.318634 809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:34.319876 809 hybrid_clock.cc:648] HybridClock initialized: now 1786389994319814 us; error 27 us; skew 500 ppm
I20260810 19:26:34.320138 809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:34 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:34.323180 809 init.cc:377] Logged in from keytab as oryx/127.19.68.2@KRBTEST.COM (short username oryx)
I20260810 19:26:34.324540 809 webserver.cc:460] Webserver started at http://127.19.68.2:38489/ using document root <none> and password file <none>
I20260810 19:26:34.325240 809 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:34.325345 809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:34.325660 809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:34.328469 809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "5b40f672c95246c0bb233d4f96315301"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "7ca18f276a9b60154d924aa74d5c2f42"
server_key_iv: "1a9f9d8b29f3eb26d8d0b7b8438966dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.329278 809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "5b40f672c95246c0bb233d4f96315301"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "7ca18f276a9b60154d924aa74d5c2f42"
server_key_iv: "1a9f9d8b29f3eb26d8d0b7b8438966dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.334455 809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:26:34.337924 838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:34.339154 809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:26:34.339303 809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "5b40f672c95246c0bb233d4f96315301"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "7ca18f276a9b60154d924aa74d5c2f42"
server_key_iv: "1a9f9d8b29f3eb26d8d0b7b8438966dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.339519 809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:34.366725 809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:34.370440 809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:34.370689 809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:34.373232 809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:34.373319 809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:34.373463 809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:34.373523 809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:26:34 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:34.396136 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.373973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:37001 (local address 127.19.68.62:44569)
0810 19:26:34.374162 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:34.374168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:34.375563 (+ 1395us) server_negotiation.cc:407] Connection header received
0810 19:26:34.378418 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:34.378427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:34.378538 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:34.378612 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:34.381351 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.382283 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.384773 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.385406 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.385536 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.389499 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:34.389503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:34.389506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:34.389541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:34.392565 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.393598 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.393604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.393608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.393778 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.394212 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.394218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.394222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.394638 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:34.394821 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:34.395478 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 19:26:34.395734 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:26:34.397205 853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.374524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.2:37001)
0810 19:26:34.375384 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:34.375439 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:34.378066 (+ 2627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:34.379197 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:34.379208 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:34.380073 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:34.381133 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.381163 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.382450 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.382457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.384578 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.384614 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.385680 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.385692 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.386036 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.386857 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:34.386896 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:34.389281 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:34.392714 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.392726 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.392743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.393446 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.393905 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.393910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.393914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.394106 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.394792 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:34.394812 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:34.395134 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 19:26:34.395357 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 19:26:34.395738 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":336,"threads_started":1}
I20260810 19:26:34.417738 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.402846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:57451 (local address 127.19.68.62:44569)
0810 19:26:34.403034 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:34.403040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:34.403688 (+ 648us) server_negotiation.cc:407] Connection header received
0810 19:26:34.403844 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:34.403847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:34.403904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:34.403977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:34.405241 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.406343 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.408554 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.409206 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.409333 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.410884 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:34.410891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:34.410915 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:34.410949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:34.414377 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.415321 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.415326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.415329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.415490 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.415954 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.415960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.415963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.416442 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:34.416598 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:34.417335 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 19:26:34.417520 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:26:34.418011 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.402852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.2:57451)
0810 19:26:34.403584 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:34.403610 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:34.403748 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:34.404056 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:34.404062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:34.404645 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:34.405041 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.405053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.406534 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.406539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.408359 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.408379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.409376 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.409381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.409515 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.410047 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:34.410063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:34.410724 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:34.414526 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.414532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.414539 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.415134 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.415627 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.415630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.415633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.415814 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.416670 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:34.416675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:34.416943 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 19:26:34.417215 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 19:26:34.417541 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":367,"threads_started":1}
I20260810 19:26:34.426852 809 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:46847
I20260810 19:26:34.426882 1050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:46847 every 8 connection(s)
I20260810 19:26:34.427978 809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:34.429750 1051 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44569
I20260810 19:26:34.429950 1051 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:34.430549 1051 heartbeater.cc:511] Master 127.19.68.62:44569 requested a full tablet report, sending...
I20260810 19:26:34.431859 429 ts_manager.cc:194] Registered new tserver with Master: 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847)
I20260810 19:26:34.432663 429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:37001
I20260810 19:26:34.435629 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 809
I20260810 19:26:34.435809 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:34.436313 19728 external_mini_cluster.cc:1428] Setting key 568ba50d40b14a3f67b8608d67760568
WARNING: no policy specified for oryx/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
I20260810 19:26:34.493638 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44569
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:34.820106 1057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:34.820374 1057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:34.820508 1057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:34.827803 1057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:34.827914 1057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:34.828039 1057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:26:34.828141 1057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:34.838165 1057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.68.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:34.839434 1057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:34.840795 1057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:34.853659 1067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:34.853744 1064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:34.853721 1065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:34.854816 1057 server_base.cc:1034] running on GCE node
I20260810 19:26:34.855366 1057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:34.856269 1057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:34.857484 1057 hybrid_clock.cc:648] HybridClock initialized: now 1786389994857419 us; error 37 us; skew 500 ppm
I20260810 19:26:34.857759 1057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:34 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:34.860863 1057 init.cc:377] Logged in from keytab as oryx/127.19.68.3@KRBTEST.COM (short username oryx)
I20260810 19:26:34.862391 1057 webserver.cc:460] Webserver started at http://127.19.68.3:44633/ using document root <none> and password file <none>
I20260810 19:26:34.863056 1057 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:34.863155 1057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:34.863473 1057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:34.867035 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "732e68b58e084760840cd645eaa957e4"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "9bb9f0978cbf2a4c1861f7f5d2349713"
server_key_iv: "26bba58bcff9766a1e6d651c37884941"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.868116 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "732e68b58e084760840cd645eaa957e4"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "9bb9f0978cbf2a4c1861f7f5d2349713"
server_key_iv: "26bba58bcff9766a1e6d651c37884941"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.873420 1057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:26:34.876740 1075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:34.878037 1057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:34.878180 1057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "732e68b58e084760840cd645eaa957e4"
format_stamp: "Formatted at 2026-08-10 19:26:34 on dist-test-slave-2j7r"
server_key: "9bb9f0978cbf2a4c1861f7f5d2349713"
server_key_iv: "26bba58bcff9766a1e6d651c37884941"
server_key_version: "encryptionkey@0"
I20260810 19:26:34.878372 1057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:34.894891 804 heartbeater.cc:503] Master 127.19.68.62:44569 was elected leader, sending a full tablet report...
I20260810 19:26:34.904616 1057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:34.908686 1057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:34.908948 1057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:34.911708 1057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:34.911806 1057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:34.911962 1057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:34.912086 1057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 19:26:34 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:34.935405 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.912237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:38861 (local address 127.19.68.62:44569)
0810 19:26:34.912469 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:34.912477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:34.913896 (+ 1419us) server_negotiation.cc:407] Connection header received
0810 19:26:34.916692 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:34.916698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:34.916788 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:34.916897 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:34.919550 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.920337 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.923193 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.923736 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.923880 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.928385 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:34.928390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:34.928393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:34.928427 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:34.931646 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.932622 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.932628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.932632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.932793 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.933385 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.933389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.933391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.933746 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:34.933895 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:34.934714 (+ 819us) server_negotiation.cc:299] Negotiation successful
0810 19:26:34.934991 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:26:34.936061 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.912829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.3:38861)
0810 19:26:34.913704 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:34.913770 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:34.916320 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:34.917349 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:34.917365 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:34.918166 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:34.919318 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.919349 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.920507 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.920514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.923037 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.923055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.924144 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.924149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.924330 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.925326 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:34.925361 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:34.928164 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:34.931795 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.931808 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.931824 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.932404 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.932936 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.932941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.932944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.933262 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.933963 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:34.933968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:34.934308 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 19:26:34.934527 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 19:26:34.934918 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":359,"threads_started":1}
I20260810 19:26:34.960533 1057 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:34813
I20260810 19:26:34.960566 1313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:34813 every 8 connection(s)
I20260810 19:26:34.961812 1057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:34.963904 1314 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44569
I20260810 19:26:34.964113 1314 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:34.964797 1314 heartbeater.cc:511] Master 127.19.68.62:44569 requested a full tablet report, sending...
I20260810 19:26:34.966135 429 ts_manager.cc:194] Registered new tserver with Master: 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:34.967192 429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:38861
I20260810 19:26:34.968254 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.954972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:53995 (local address 127.19.68.62:44569)
0810 19:26:34.955129 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:34.955135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:34.955261 (+ 126us) server_negotiation.cc:407] Connection header received
0810 19:26:34.955449 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:34.955454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:34.955555 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:34.955658 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:34.956991 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.958106 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.959649 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.960275 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.960401 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.962436 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:34.962443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:34.962448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:34.962500 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:34.965649 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.966343 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.966348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.966351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.966531 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:34.966880 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:34.966885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:34.966888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:34.967251 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:34.967405 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:34.967771 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 19:26:34.968008 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":61}
I20260810 19:26:34.968389 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:34.954449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.19.68.3:53995)
0810 19:26:34.955129 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:34.955148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:34.955301 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:34.955718 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:34.955723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:34.956392 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:34.956827 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.956840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.958239 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.958245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.959509 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:34.959523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:34.960517 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:34.960524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:34.960721 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:34.961416 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:34.961433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:34.962216 (+ 783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:34.965783 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.965788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.965792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.966188 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.966635 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:34.966638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:34.966640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:34.966782 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:34.967386 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:34.967391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:34.967544 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 19:26:34.967660 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 19:26:34.967897 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":678,"threads_started":1}
I20260810 19:26:34.970709 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 1057
I20260810 19:26:34.970886 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:34.971405 19728 external_mini_cluster.cc:1428] Setting key b193dabda6950066324bdddff81ebd39
I20260810 19:26:34.974268 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:26:35 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:26:35 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:35 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:26:35.032971 596 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:35.022534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.19.68.62:44569)
0810 19:26:35.022781 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.022787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.022982 (+ 195us) server_negotiation.cc:407] Connection header received
0810 19:26:35.023182 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.023188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.023287 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.023404 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:35.024797 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.025670 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.026944 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.027366 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.027479 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.032655 (+ 5176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44474: BlockingRecv error: recv got EOF from 127.0.0.1:44474 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 10 19:26:35 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:35.049754 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.035989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.19.68.62:44569)
0810 19:26:35.036256 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.036263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.036551 (+ 288us) server_negotiation.cc:407] Connection header received
0810 19:26:35.036718 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.036725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.036799 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.036884 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:35.038180 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.038918 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.040215 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.040627 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.040750 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.044516 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:35.044521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:35.044525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:35.044560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:35.047094 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:35.047768 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:35.047774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:35.047778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:35.047926 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:35.048314 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:35.048319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:35.048321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:35.048616 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:35.048913 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.049199 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.049445 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:26:35.053694 429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:35.060635 429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:35.083411 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.073730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.19.68.3:34813)
0810 19:26:35.074555 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.074564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.075294 (+ 730us) server_negotiation.cc:407] Connection header received
0810 19:26:35.075404 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.075412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.075677 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.075868 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:35.076732 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.078396 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.080705 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.082124 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.082256 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.082504 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.082887 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.083055 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":481,"threads_started":1}
I20260810 19:26:35.083454 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.074000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46847 (local address 127.0.0.1:55992)
0810 19:26:35.076016 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.076032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.076142 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.076710 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.076714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.076739 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:35.077112 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.077127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.078775 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.078781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.080697 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.080713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.082662 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.082669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.082821 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.082866 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.082950 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.083068 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1910,"mutex_wait_us":4,"thread_start_us":1773,"threads_started":1}
I20260810 19:26:35.083664 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.074554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.19.68.2:46847)
0810 19:26:35.075610 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.075619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.076203 (+ 584us) server_negotiation.cc:407] Connection header received
0810 19:26:35.076314 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.076322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.076547 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.076718 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:35.077328 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.078650 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.081017 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.082399 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.082581 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.082773 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.083204 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.083380 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":774,"threads_started":1}
I20260810 19:26:35.083968 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.073401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:34813 (local address 127.0.0.1:49116)
0810 19:26:35.075030 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.075049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.075175 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.076016 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.076024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.076057 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:35.076509 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.076523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.078534 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.078539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.080536 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.080549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.082306 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.082312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.082504 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.082560 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.082688 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.082875 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":1317,"threads_started":1}
I20260810 19:26:35.084497 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.072680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39617 (local address 127.0.0.1:56404)
0810 19:26:35.073656 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.073708 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.073899 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.077483 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.077488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.077513 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:35.077948 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.077960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.079863 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.079868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.081857 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.081871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.083813 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.083820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.083999 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.084026 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.084096 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.084220 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":1,"thread_start_us":534,"threads_started":1}
I20260810 19:26:35.085212 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.072991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.19.68.1:39617)
0810 19:26:35.076737 (+ 3746us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.076745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.076771 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:26:35.076872 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.076878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.077116 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.078135 (+ 1019us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:35.078342 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.079713 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.081996 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.083572 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.084352 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.084554 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.084760 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.084933 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":3358,"threads_started":1}
I20260810 19:26:35.087930 676 tablet_service.cc:1467] Processing CreateTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec (DEFAULT_TABLE table=test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.087767 927 tablet_service.cc:1467] Processing CreateTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec (DEFAULT_TABLE table=test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.088056 1181 tablet_service.cc:1467] Processing CreateTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec (DEFAULT_TABLE table=test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.089130 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da00f5f38aab47f391ce1c8b4e8aa8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.089210 927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da00f5f38aab47f391ce1c8b4e8aa8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.089257 1181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da00f5f38aab47f391ce1c8b4e8aa8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.100373 1359 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Bootstrap starting.
I20260810 19:26:35.103312 1362 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Bootstrap starting.
I20260810 19:26:35.104090 1361 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Bootstrap starting.
I20260810 19:26:35.104817 1359 tablet_bootstrap.cc:654] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.106189 1359 log.cc:824] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:35.107402 1361 tablet_bootstrap.cc:654] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.107885 1362 tablet_bootstrap.cc:654] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.108587 1361 log.cc:824] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:35.108806 1362 log.cc:824] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:35.109524 1359 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: No bootstrap required, opened a new log
I20260810 19:26:35.109818 1359 ts_tablet_manager.cc:1397] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 19:26:35.111487 1361 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: No bootstrap required, opened a new log
I20260810 19:26:35.111670 1362 tablet_bootstrap.cc:492] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: No bootstrap required, opened a new log
I20260810 19:26:35.111747 1361 ts_tablet_manager.cc:1397] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 19:26:35.112322 1362 ts_tablet_manager.cc:1397] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 19:26:35.117514 1359 raft_consensus.cc:348] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.117871 1359 raft_consensus.cc:374] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.117995 1359 raft_consensus.cc:729] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328810c709c84794889c6b3d12221d27, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.118553 1359 consensus_queue.cc:260] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.120173 1359 ts_tablet_manager.cc:1428] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260810 19:26:35.121940 1361 raft_consensus.cc:348] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.122386 1361 raft_consensus.cc:374] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.122496 1361 raft_consensus.cc:729] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.122445 1362 raft_consensus.cc:348] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.122911 1362 raft_consensus.cc:374] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.123050 1362 raft_consensus.cc:729] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.123243 1361 consensus_queue.cc:260] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.123762 1362 consensus_queue.cc:260] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.124462 1051 heartbeater.cc:503] Master 127.19.68.62:44569 was elected leader, sending a full tablet report...
I20260810 19:26:35.125174 1361 ts_tablet_manager.cc:1428] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260810 19:26:35.129793 1314 heartbeater.cc:503] Master 127.19.68.62:44569 was elected leader, sending a full tablet report...
I20260810 19:26:35.130290 1362 ts_tablet_manager.cc:1428] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
W20260810 19:26:35.136909 805 tablet.cc:2369] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:26:35.179952 1052 tablet.cc:2369] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:26:35.213492 1316 tablet.cc:2369] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:26:35.294862 1379 raft_consensus.cc:482] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:35.295198 1379 raft_consensus.cc:504] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.296984 1379 leader_election.cc:288] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.306195 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.297381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:60467 (local address 127.19.68.3:34813)
0810 19:26:35.297637 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.297644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.298601 (+ 957us) server_negotiation.cc:407] Connection header received
0810 19:26:35.298768 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.298772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.298840 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.298907 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:35.299599 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.300688 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.303632 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.304908 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.305052 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.305202 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.305754 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.305903 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 19:26:35.306350 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.297293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39617 (local address 127.19.68.2:39377)
0810 19:26:35.298072 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.298088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.298255 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.298651 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.298656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.298678 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:35.299126 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.299140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.300770 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.300777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.303465 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.303493 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.305216 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.305220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.305493 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.305572 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.305671 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.305862 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":254,"thread_start_us":435,"threads_started":1}
I20260810 19:26:35.306416 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.297800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:34813 (local address 127.19.68.2:60467)
0810 19:26:35.298500 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.298514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.298644 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.298985 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.298990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.299014 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:35.299395 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.299406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.300843 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.300847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.303460 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.303478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.305196 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.305207 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.305492 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.305573 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.305671 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.305862 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":214,"thread_start_us":369,"threads_started":1}
I20260810 19:26:35.306857 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.297491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:39377 (local address 127.19.68.1:39617)
0810 19:26:35.297805 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.297811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.298190 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:26:35.298442 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.298448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.298528 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.298614 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:35.299381 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.300593 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.303654 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.304957 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.305114 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.305213 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.305753 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.305903 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260810 19:26:35.307534 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da00f5f38aab47f391ce1c8b4e8aa8ec" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4" is_pre_election: true
I20260810 19:26:35.307869 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da00f5f38aab47f391ce1c8b4e8aa8ec" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27" is_pre_election: true
I20260810 19:26:35.308117 1215 raft_consensus.cc:2454] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.308485 722 raft_consensus.cc:2454] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.308826 839 leader_election.cc:302] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b40f672c95246c0bb233d4f96315301, 732e68b58e084760840cd645eaa957e4; no voters:
I20260810 19:26:35.309288 1379 raft_consensus.cc:2781] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:35.309429 1379 raft_consensus.cc:482] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:35.309541 1379 raft_consensus.cc:3037] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.311852 1379 raft_consensus.cc:504] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.312587 1379 leader_election.cc:288] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Requested vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.312968 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da00f5f38aab47f391ce1c8b4e8aa8ec" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27"
I20260810 19:26:35.313069 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da00f5f38aab47f391ce1c8b4e8aa8ec" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4"
I20260810 19:26:35.313246 722 raft_consensus.cc:3037] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.313300 1215 raft_consensus.cc:3037] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.316596 1215 raft_consensus.cc:2454] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.316597 722 raft_consensus.cc:2454] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.317025 839 leader_election.cc:302] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301, 732e68b58e084760840cd645eaa957e4; no voters:
I20260810 19:26:35.317418 1379 raft_consensus.cc:2781] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:35.317862 1379 raft_consensus.cc:686] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Becoming Leader. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Running, Role: LEADER
I20260810 19:26:35.318315 1379 consensus_queue.cc:237] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.323010 429 catalog_manager.cc:5495] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b40f672c95246c0bb233d4f96315301 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:35.377527 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.372438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.19.68.2:46847)
0810 19:26:35.372642 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.372648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.372668 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:26:35.372863 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.372867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.372964 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.373120 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:35.373506 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.374212 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.375499 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.375894 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.376006 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.376421 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:35.376934 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:35.377066 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.377194 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.377276 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:26:35.385416 1215 raft_consensus.cc:1261] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.385432 722 raft_consensus.cc:1261] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.386349 1388 consensus_queue.cc:1035] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.386729 1379 consensus_queue.cc:1035] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.397886 1396 mvcc.cc:205] Tried to move back new op lower bound from 7317053421088927744 to 7317053420827521024. Current Snapshot: MvccSnapshot[applied={T|T < 7317053421088927744}]
I20260810 19:26:35.415120 428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:35.415459 428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:35.419296 428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Sending DeleteTablet for 3 replicas of tablet da00f5f38aab47f391ce1c8b4e8aa8ec
I20260810 19:26:35.420358 676 tablet_service.cc:1514] Processing DeleteTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:56404
I20260810 19:26:35.420472 927 tablet_service.cc:1514] Processing DeleteTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55992
I20260810 19:26:35.420740 1181 tablet_service.cc:1514] Processing DeleteTablet for tablet da00f5f38aab47f391ce1c8b4e8aa8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:35.421275 1412 tablet_replica.cc:330] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: stopping tablet replica
I20260810 19:26:35.421713 1412 raft_consensus.cc:2229] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:35.422282 1412 raft_consensus.cc:2258] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.423094 428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:35.424418 428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:35.424491 1412 ts_tablet_manager.cc:1905] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.430320 1417 tablet_replica.cc:330] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: stopping tablet replica
I20260810 19:26:35.431455 1417 raft_consensus.cc:2229] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:35.431874 1412 ts_tablet_manager.cc:1918] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:35.432054 1412 log.cc:1196] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/da00f5f38aab47f391ce1c8b4e8aa8ec
I20260810 19:26:35.432551 1412 ts_tablet_manager.cc:1939] T da00f5f38aab47f391ce1c8b4e8aa8ec P 328810c709c84794889c6b3d12221d27: Deleting consensus metadata
I20260810 19:26:35.432578 1417 raft_consensus.cc:2258] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.434849 407 catalog_manager.cc:4841] TS 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617): tablet da00f5f38aab47f391ce1c8b4e8aa8ec (table test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]) successfully deleted
I20260810 19:26:35.435223 1180 tablet_service.cc:1467] Processing CreateTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 (DEFAULT_TABLE table=test-table [id=c849227576d04269bf37d9f725cc23b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.435670 1180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1944d7d0f4c49a18544b0f17a1a1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.435817 926 tablet_service.cc:1467] Processing CreateTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 (DEFAULT_TABLE table=test-table [id=c849227576d04269bf37d9f725cc23b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.436473 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1944d7d0f4c49a18544b0f17a1a1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.435693 1417 ts_tablet_manager.cc:1905] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.441708 676 tablet_service.cc:1467] Processing CreateTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 (DEFAULT_TABLE table=test-table [id=c849227576d04269bf37d9f725cc23b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:35.442139 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1944d7d0f4c49a18544b0f17a1a1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.445372 1362 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Bootstrap starting.
I20260810 19:26:35.447371 1362 tablet_bootstrap.cc:654] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.451792 1417 ts_tablet_manager.cc:1918] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:35.452186 1417 log.cc:1196] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/da00f5f38aab47f391ce1c8b4e8aa8ec
I20260810 19:26:35.452975 1417 ts_tablet_manager.cc:1939] T da00f5f38aab47f391ce1c8b4e8aa8ec P 732e68b58e084760840cd645eaa957e4: Deleting consensus metadata
I20260810 19:26:35.455389 403 catalog_manager.cc:4841] TS 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813): tablet da00f5f38aab47f391ce1c8b4e8aa8ec (table test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]) successfully deleted
I20260810 19:26:35.458940 1359 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Bootstrap starting.
I20260810 19:26:35.459493 1415 tablet_replica.cc:330] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: stopping tablet replica
I20260810 19:26:35.460642 1359 tablet_bootstrap.cc:654] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.460745 1362 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: No bootstrap required, opened a new log
I20260810 19:26:35.460891 1362 ts_tablet_manager.cc:1397] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20260810 19:26:35.461743 1362 raft_consensus.cc:348] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.462090 1362 raft_consensus.cc:374] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.460865 1361 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Bootstrap starting.
I20260810 19:26:35.462291 1362 raft_consensus.cc:729] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.462647 1362 consensus_queue.cc:260] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.463559 1359 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: No bootstrap required, opened a new log
I20260810 19:26:35.463755 1359 ts_tablet_manager.cc:1397] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:26:35.463903 1361 tablet_bootstrap.cc:654] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.464290 1362 ts_tablet_manager.cc:1428] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 19:26:35.464907 1415 raft_consensus.cc:2229] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:35.465687 1415 raft_consensus.cc:2258] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.466501 1359 raft_consensus.cc:348] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.467164 1359 raft_consensus.cc:374] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.467312 1359 raft_consensus.cc:729] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328810c709c84794889c6b3d12221d27, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.467664 1359 consensus_queue.cc:260] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.470563 1361 tablet_bootstrap.cc:492] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: No bootstrap required, opened a new log
I20260810 19:26:35.470671 1415 ts_tablet_manager.cc:1905] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.470764 1361 ts_tablet_manager.cc:1397] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260810 19:26:35.471374 1359 ts_tablet_manager.cc:1428] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260810 19:26:35.471659 1361 raft_consensus.cc:348] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.471927 1361 raft_consensus.cc:374] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.472074 1361 raft_consensus.cc:729] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.472394 1361 consensus_queue.cc:260] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.475013 1361 ts_tablet_manager.cc:1428] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260810 19:26:35.478008 1415 ts_tablet_manager.cc:1918] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:35.478173 1415 log.cc:1196] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/da00f5f38aab47f391ce1c8b4e8aa8ec
I20260810 19:26:35.478667 1415 ts_tablet_manager.cc:1939] T da00f5f38aab47f391ce1c8b4e8aa8ec P 5b40f672c95246c0bb233d4f96315301: Deleting consensus metadata
I20260810 19:26:35.480227 407 catalog_manager.cc:4841] TS 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847): tablet da00f5f38aab47f391ce1c8b4e8aa8ec (table test-table [id=8c7ebfcadae5409ca8921cb38f0fd4e4]) successfully deleted
I20260810 19:26:35.491487 1388 raft_consensus.cc:482] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:35.491676 1388 raft_consensus.cc:504] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.492314 1388 leader_election.cc:288] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.492832 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1944d7d0f4c49a18544b0f17a1a1d54" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27" is_pre_election: true
I20260810 19:26:35.492998 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1944d7d0f4c49a18544b0f17a1a1d54" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4" is_pre_election: true
I20260810 19:26:35.493132 722 raft_consensus.cc:2454] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.493289 1215 raft_consensus.cc:2454] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.493577 843 leader_election.cc:302] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:35.493889 1388 raft_consensus.cc:2781] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:35.493991 1388 raft_consensus.cc:482] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:35.494099 1388 raft_consensus.cc:3037] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.496166 1388 raft_consensus.cc:504] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.496695 1388 leader_election.cc:288] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Requested vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.497177 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1944d7d0f4c49a18544b0f17a1a1d54" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27"
I20260810 19:26:35.497187 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1944d7d0f4c49a18544b0f17a1a1d54" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4"
I20260810 19:26:35.497346 722 raft_consensus.cc:3037] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.497406 1215 raft_consensus.cc:3037] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.499409 722 raft_consensus.cc:2454] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.499780 843 leader_election.cc:302] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:35.500047 1388 raft_consensus.cc:2781] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:35.500267 1215 raft_consensus.cc:2454] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.500279 1388 raft_consensus.cc:686] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Becoming Leader. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Running, Role: LEADER
I20260810 19:26:35.500555 1388 consensus_queue.cc:237] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.510033 428 catalog_manager.cc:5495] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b40f672c95246c0bb233d4f96315301 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } health_report { overall_health: UNKNOWN } } }
Aug 10 19:26:35 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389993, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.62@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:35.572544 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.555294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44569 (local address 127.0.0.1:44500)
0810 19:26:35.556108 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.556123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.556257 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.556564 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.556569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.557357 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:35.557670 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.557682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.558828 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.558833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.559791 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.559804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.560511 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.560518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.560668 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.564794 (+ 4126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:35.564812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:35.567238 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:35.570101 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:35.570114 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:35.570122 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:35.570525 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:35.570859 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:35.570864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:35.570868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:35.571013 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:35.571515 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:35.571522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:35.571684 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.571829 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.572092 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":523,"threads_started":1}
I20260810 19:26:35.572585 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.555297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.19.68.62:44569)
0810 19:26:35.555947 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.555953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.556202 (+ 249us) server_negotiation.cc:407] Connection header received
0810 19:26:35.556364 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.556368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.556437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.556524 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:35.557828 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.558712 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.559925 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.560323 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.560442 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.567436 (+ 6994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:35.567440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:35.567443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:35.567482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:35.569983 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:35.570630 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:35.570635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:35.570638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:35.570758 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:35.571111 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:35.571116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:35.571119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:35.571380 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:35.571534 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.571925 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.572223 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260810 19:26:35.577836 428 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:44500:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:26:35.578819 428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:35.586505 1181 tablet_service.cc:1467] Processing CreateTablet for tablet 6e863e7bc4bf4d128ed71394fd8a539d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902b1a60adf24fbaa7085e5cac162433]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:26:35.586720 676 tablet_service.cc:1467] Processing CreateTablet for tablet 6e863e7bc4bf4d128ed71394fd8a539d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902b1a60adf24fbaa7085e5cac162433]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:26:35.587141 1181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e863e7bc4bf4d128ed71394fd8a539d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.587226 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e863e7bc4bf4d128ed71394fd8a539d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.589717 926 tablet_service.cc:1467] Processing CreateTablet for tablet 6e863e7bc4bf4d128ed71394fd8a539d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902b1a60adf24fbaa7085e5cac162433]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:26:35.590288 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e863e7bc4bf4d128ed71394fd8a539d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:35.594271 1362 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: Bootstrap starting.
I20260810 19:26:35.594381 1359 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: Bootstrap starting.
I20260810 19:26:35.595685 1359 tablet_bootstrap.cc:654] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.596084 1362 tablet_bootstrap.cc:654] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.597819 1359 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: No bootstrap required, opened a new log
I20260810 19:26:35.597973 1359 ts_tablet_manager.cc:1397] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 19:26:35.598621 1359 raft_consensus.cc:348] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.598829 1359 raft_consensus.cc:374] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.598912 1359 raft_consensus.cc:729] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328810c709c84794889c6b3d12221d27, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.599182 1359 consensus_queue.cc:260] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.599584 1375 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.599953 1375 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.600586 1359 ts_tablet_manager.cc:1428] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:35.601444 1361 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: Bootstrap starting.
I20260810 19:26:35.603406 1361 tablet_bootstrap.cc:654] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:35.606487 1361 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: No bootstrap required, opened a new log
I20260810 19:26:35.606676 1361 ts_tablet_manager.cc:1397] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260810 19:26:35.607503 1361 raft_consensus.cc:348] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.607787 1361 raft_consensus.cc:374] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.607925 1361 raft_consensus.cc:729] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.608243 1361 consensus_queue.cc:260] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.608745 1388 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.609001 1388 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.609630 1361 ts_tablet_manager.cc:1428] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:35.614437 1362 tablet_bootstrap.cc:492] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: No bootstrap required, opened a new log
I20260810 19:26:35.614598 1362 ts_tablet_manager.cc:1397] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.009s
I20260810 19:26:35.615160 1362 raft_consensus.cc:348] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.615410 1362 raft_consensus.cc:374] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:35.615486 1362 raft_consensus.cc:729] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Initialized, Role: FOLLOWER
I20260810 19:26:35.615720 1362 consensus_queue.cc:260] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.616186 1362 ts_tablet_manager.cc:1428] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 19:26:35.616181 1383 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.616591 1383 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.761960 1388 raft_consensus.cc:482] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:35.762164 1388 raft_consensus.cc:504] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.762812 1388 leader_election.cc:288] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.763325 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e863e7bc4bf4d128ed71394fd8a539d" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27" is_pre_election: true
I20260810 19:26:35.763473 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e863e7bc4bf4d128ed71394fd8a539d" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4" is_pre_election: true
I20260810 19:26:35.763624 722 raft_consensus.cc:2454] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.763736 1215 raft_consensus.cc:2454] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:35.764101 843 leader_election.cc:302] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:35.764392 1388 raft_consensus.cc:2781] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:35.764528 1388 raft_consensus.cc:482] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:35.764626 1388 raft_consensus.cc:3037] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.767076 1388 raft_consensus.cc:504] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.767609 1388 leader_election.cc:288] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Requested vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:35.768035 722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e863e7bc4bf4d128ed71394fd8a539d" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27"
I20260810 19:26:35.768090 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e863e7bc4bf4d128ed71394fd8a539d" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4"
I20260810 19:26:35.768263 722 raft_consensus.cc:3037] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.768286 1215 raft_consensus.cc:3037] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:35.771121 722 raft_consensus.cc:2454] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.771121 1215 raft_consensus.cc:2454] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:35.771508 843 leader_election.cc:302] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:35.771844 1388 raft_consensus.cc:2781] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:35.771996 1388 raft_consensus.cc:686] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Becoming Leader. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Running, Role: LEADER
I20260810 19:26:35.772276 1388 consensus_queue.cc:237] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:35.773852 1404 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: TxnStatusTablet state changed. Reason: New leader 5b40f672c95246c0bb233d4f96315301. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.774253 1404 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:26:35.774978 1447 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:26:35.775184 1447 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:26:35.775533 428 catalog_manager.cc:5495] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b40f672c95246c0bb233d4f96315301 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:35.805584 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.800579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46847 (local address 127.0.0.1:56016)
0810 19:26:35.800796 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.800817 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.800952 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.801338 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.801342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.801354 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:26:35.801541 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.801549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.802555 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.802560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.803657 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.803670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.804418 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.804424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.804566 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.804620 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:26:35.805035 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:26:35.805056 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.805148 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.805294 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:26:35.805624 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.800704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.19.68.2:46847)
0810 19:26:35.800885 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.800890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.800921 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:26:35.801153 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.801157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.801223 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.801303 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:35.801671 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.802421 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.803811 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.804259 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.804375 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.804805 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:35.804936 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:35.805019 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.805188 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.805267 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:26:35.809449 1215 raft_consensus.cc:1261] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.809494 722 raft_consensus.cc:1261] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.810128 1388 consensus_queue.cc:1035] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.810487 1404 consensus_queue.cc:1035] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.813827 1375 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: TxnStatusTablet state changed. Reason: New leader 5b40f672c95246c0bb233d4f96315301. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.814095 1375 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.815970 1404 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.816246 1404 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:26:35.816535 1393 mvcc.cc:205] Tried to move back new op lower bound from 7317053422829395968 to 7317053422685147136. Current Snapshot: MvccSnapshot[applied={T|T < 7317053422829395968 or (T in {7317053422829395968})}]
I20260810 19:26:35.819169 1375 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.819393 1375 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 328810c709c84794889c6b3d12221d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.824721 1383 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: TxnStatusTablet state changed. Reason: New leader 5b40f672c95246c0bb233d4f96315301. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.824993 1383 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.825645 1379 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.825968 1379 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 5b40f672c95246c0bb233d4f96315301: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:26:35.829300 1383 tablet_replica.cc:439] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } }
I20260810 19:26:35.829528 1383 tablet_replica.cc:442] T 6e863e7bc4bf4d128ed71394fd8a539d P 732e68b58e084760840cd645eaa957e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:26:35.846627 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.838776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:60569 (local address 127.19.68.2:46847)
0810 19:26:35.838916 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:35.838920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:35.841618 (+ 2698us) server_negotiation.cc:407] Connection header received
0810 19:26:35.841787 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:35.841792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:35.841854 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:35.841930 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:35.842383 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.843284 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.844790 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.845300 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.845414 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.845865 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:35.845991 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:35.846084 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:35.846244 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 19:26:35.846330 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260810 19:26:35.846709 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:35.838704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46847 (local address 127.19.68.2:60569)
0810 19:26:35.841520 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:35.841535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:35.841672 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:35.841941 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:35.841945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:35.841955 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:26:35.842232 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.842244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.843407 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.843412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.844665 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:35.844680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:35.845496 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:35.845501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:35.845638 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:35.845690 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:26:35.846082 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:26:35.846094 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 19:26:35.846195 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 19:26:35.846326 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"thread_start_us":2549,"threads_started":1}
I20260810 19:26:35.856724 722 raft_consensus.cc:1261] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.856751 1215 raft_consensus.cc:1261] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:35.857251 1388 consensus_queue.cc:1035] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.857470 1404 consensus_queue.cc:1035] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:35.946979 427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:35.947295 427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:35.951273 427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Sending DeleteTablet for 3 replicas of tablet c1944d7d0f4c49a18544b0f17a1a1d54
I20260810 19:26:35.952112 676 tablet_service.cc:1514] Processing DeleteTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:56404
I20260810 19:26:35.952320 926 tablet_service.cc:1514] Processing DeleteTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55992
I20260810 19:26:35.952629 1415 tablet_replica.cc:330] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: stopping tablet replica
I20260810 19:26:35.952888 1415 raft_consensus.cc:2229] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:35.953295 1415 raft_consensus.cc:2258] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.953619 1181 tablet_service.cc:1514] Processing DeleteTablet for tablet c1944d7d0f4c49a18544b0f17a1a1d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:35 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:35.953914 1417 tablet_replica.cc:330] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: stopping tablet replica
I20260810 19:26:35.954111 1417 raft_consensus.cc:2229] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:35.954303 1417 raft_consensus.cc:2258] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.955917 1415 ts_tablet_manager.cc:1905] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.963632 1417 ts_tablet_manager.cc:1905] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.963836 1415 ts_tablet_manager.cc:1918] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:35.964012 1415 log.cc:1196] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/c1944d7d0f4c49a18544b0f17a1a1d54
I20260810 19:26:35.964427 1415 ts_tablet_manager.cc:1939] T c1944d7d0f4c49a18544b0f17a1a1d54 P 5b40f672c95246c0bb233d4f96315301: Deleting consensus metadata
I20260810 19:26:35.968806 407 catalog_manager.cc:4841] TS 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847): tablet c1944d7d0f4c49a18544b0f17a1a1d54 (table test-table [id=c849227576d04269bf37d9f725cc23b9]) successfully deleted
I20260810 19:26:35.971436 1417 ts_tablet_manager.cc:1918] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:35.971961 1417 log.cc:1196] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/c1944d7d0f4c49a18544b0f17a1a1d54
I20260810 19:26:35.972462 1417 ts_tablet_manager.cc:1939] T c1944d7d0f4c49a18544b0f17a1a1d54 P 732e68b58e084760840cd645eaa957e4: Deleting consensus metadata
I20260810 19:26:35.973620 403 catalog_manager.cc:4841] TS 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813): tablet c1944d7d0f4c49a18544b0f17a1a1d54 (table test-table [id=c849227576d04269bf37d9f725cc23b9]) successfully deleted
I20260810 19:26:35.976534 1460 tablet_replica.cc:330] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: stopping tablet replica
I20260810 19:26:35.976697 1460 raft_consensus.cc:2229] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:35.976876 1460 raft_consensus.cc:2258] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:35.977949 1460 ts_tablet_manager.cc:1905] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:35.984874 1460 ts_tablet_manager.cc:1918] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:35.985015 1460 log.cc:1196] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/c1944d7d0f4c49a18544b0f17a1a1d54
I20260810 19:26:35.988094 1460 ts_tablet_manager.cc:1939] T c1944d7d0f4c49a18544b0f17a1a1d54 P 328810c709c84794889c6b3d12221d27: Deleting consensus metadata
I20260810 19:26:35.989904 407 catalog_manager.cc:4841] TS 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617): tablet c1944d7d0f4c49a18544b0f17a1a1d54 (table test-table [id=c849227576d04269bf37d9f725cc23b9]) successfully deleted
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:26:36.041759 1436 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:36.030664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.19.68.62:44569)
0810 19:26:36.030888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.030895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.031274 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:26:36.031492 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.031496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.031564 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.031650 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:36.033360 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.034093 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.035852 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.036262 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.036349 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.041493 (+ 5144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44508: BlockingRecv error: recv got EOF from 127.0.0.1:44508 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM
I20260810 19:26:36.058883 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.044354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.19.68.62:44569)
0810 19:26:36.044525 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.044531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.044796 (+ 265us) server_negotiation.cc:407] Connection header received
0810 19:26:36.044916 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.044921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.044998 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.045121 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:36.046211 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.047302 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.048512 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.049182 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.049303 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.053284 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:36.053290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:36.053293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:36.053332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:36.056129 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.056952 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.056956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.056958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.057132 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.057533 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.057537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.057540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.057859 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:36.058060 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.058367 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.058564 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:26:36.062824 427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:26:36.063488 427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:36.070195 676 tablet_service.cc:1467] Processing CreateTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 (DEFAULT_TABLE table=test-table [id=860970ae84bd4e178a83c2af640e71c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.070559 1181 tablet_service.cc:1467] Processing CreateTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 (DEFAULT_TABLE table=test-table [id=860970ae84bd4e178a83c2af640e71c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.070677 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24862a0b81cb4da4a9798ea3f97d3198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.070972 1181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24862a0b81cb4da4a9798ea3f97d3198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.076812 926 tablet_service.cc:1467] Processing CreateTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 (DEFAULT_TABLE table=test-table [id=860970ae84bd4e178a83c2af640e71c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.077176 1359 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Bootstrap starting.
I20260810 19:26:36.077332 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24862a0b81cb4da4a9798ea3f97d3198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.078418 1362 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Bootstrap starting.
I20260810 19:26:36.078745 1359 tablet_bootstrap.cc:654] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.080029 1362 tablet_bootstrap.cc:654] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.084293 1361 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Bootstrap starting.
I20260810 19:26:36.086081 1361 tablet_bootstrap.cc:654] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.088398 1361 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: No bootstrap required, opened a new log
I20260810 19:26:36.088390 1362 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: No bootstrap required, opened a new log
I20260810 19:26:36.088538 1362 ts_tablet_manager.cc:1397] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260810 19:26:36.088550 1361 ts_tablet_manager.cc:1397] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 19:26:36.089143 1359 tablet_bootstrap.cc:492] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: No bootstrap required, opened a new log
I20260810 19:26:36.089362 1359 ts_tablet_manager.cc:1397] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260810 19:26:36.089342 1361 raft_consensus.cc:348] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.089329 1362 raft_consensus.cc:348] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.089614 1361 raft_consensus.cc:374] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.089608 1362 raft_consensus.cc:374] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.089707 1362 raft_consensus.cc:729] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.089715 1361 raft_consensus.cc:729] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.089928 1362 consensus_queue.cc:260] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.089977 1361 consensus_queue.cc:260] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.090512 1362 ts_tablet_manager.cc:1428] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 19:26:36.091495 1359 raft_consensus.cc:348] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.091794 1359 raft_consensus.cc:374] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.091917 1359 raft_consensus.cc:729] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328810c709c84794889c6b3d12221d27, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.092175 1359 consensus_queue.cc:260] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.092834 1359 ts_tablet_manager.cc:1428] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 19:26:36.093710 1361 ts_tablet_manager.cc:1428] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260810 19:26:36.123220 1388 raft_consensus.cc:482] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:36.123456 1388 raft_consensus.cc:504] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.124085 1388 leader_election.cc:288] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:36.124647 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24862a0b81cb4da4a9798ea3f97d3198" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27" is_pre_election: true
I20260810 19:26:36.124693 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24862a0b81cb4da4a9798ea3f97d3198" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4" is_pre_election: true
I20260810 19:26:36.124876 721 raft_consensus.cc:2454] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:36.124963 1215 raft_consensus.cc:2454] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 0.
I20260810 19:26:36.125326 843 leader_election.cc:302] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:36.125625 1388 raft_consensus.cc:2781] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:36.125732 1388 raft_consensus.cc:482] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:36.125852 1388 raft_consensus.cc:3037] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.128028 1388 raft_consensus.cc:504] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.129127 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24862a0b81cb4da4a9798ea3f97d3198" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27"
I20260810 19:26:36.129321 721 raft_consensus.cc:3037] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.129249 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24862a0b81cb4da4a9798ea3f97d3198" candidate_uuid: "5b40f672c95246c0bb233d4f96315301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732e68b58e084760840cd645eaa957e4"
I20260810 19:26:36.129477 1215 raft_consensus.cc:3037] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.131383 721 raft_consensus.cc:2454] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:36.128863 1388 leader_election.cc:288] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Requested vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813)
I20260810 19:26:36.131891 843 leader_election.cc:302] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 5b40f672c95246c0bb233d4f96315301; no voters:
I20260810 19:26:36.132145 1388 raft_consensus.cc:2781] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:36.132318 1388 raft_consensus.cc:686] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Becoming Leader. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Running, Role: LEADER
I20260810 19:26:36.132328 1215 raft_consensus.cc:2454] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b40f672c95246c0bb233d4f96315301 in term 1.
I20260810 19:26:36.132594 1388 consensus_queue.cc:237] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } }
I20260810 19:26:36.135522 427 catalog_manager.cc:5495] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b40f672c95246c0bb233d4f96315301 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "5b40f672c95246c0bb233d4f96315301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:36.162832 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.157346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.19.68.2:46847)
0810 19:26:36.157526 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.157530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.157553 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:26:36.157770 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.157775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.157847 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.157959 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:36.158376 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.159116 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.160980 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.161518 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.161627 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.162116 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.162251 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.162312 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.162465 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.162537 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:26:36.165529 721 raft_consensus.cc:1261] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:36.165529 1215 raft_consensus.cc:1261] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Refusing update from remote peer 5b40f672c95246c0bb233d4f96315301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:36.166182 1379 consensus_queue.cc:1035] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:36.166591 1388 consensus_queue.cc:1035] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:36.176749 428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:36.177042 428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:36.182513 1181 tablet_service.cc:1514] Processing DeleteTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:36.182863 1417 tablet_replica.cc:330] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: stopping tablet replica
I20260810 19:26:36.183058 1417 raft_consensus.cc:2229] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:36.183293 1417 raft_consensus.cc:2258] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.185763 926 tablet_service.cc:1514] Processing DeleteTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55992
I20260810 19:26:36.186067 1415 tablet_replica.cc:330] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: stopping tablet replica
I20260810 19:26:36.186242 1415 raft_consensus.cc:2229] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:36.186334 1417 ts_tablet_manager.cc:1905] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:36.186549 1415 raft_consensus.cc:2258] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.187384 1415 ts_tablet_manager.cc:1905] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:36.191943 1415 ts_tablet_manager.cc:1918] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:36.192076 1415 log.cc:1196] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.192425 428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Sending DeleteTablet for 3 replicas of tablet 24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.192489 1415 ts_tablet_manager.cc:1939] T 24862a0b81cb4da4a9798ea3f97d3198 P 5b40f672c95246c0bb233d4f96315301: Deleting consensus metadata
I20260810 19:26:36.193324 676 tablet_service.cc:1514] Processing DeleteTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:56404
I20260810 19:26:36.193614 1460 tablet_replica.cc:330] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: stopping tablet replica
I20260810 19:26:36.193661 926 tablet_service.cc:1514] Processing DeleteTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55992
I20260810 19:26:36.193815 1460 raft_consensus.cc:2229] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:36.194070 1460 raft_consensus.cc:2258] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.194556 407 catalog_manager.cc:4841] TS 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847): tablet 24862a0b81cb4da4a9798ea3f97d3198 (table test-table [id=860970ae84bd4e178a83c2af640e71c0]) successfully deleted
W20260810 19:26:36.195021 407 catalog_manager.cc:4808] TS 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847): delete failed for tablet 24862a0b81cb4da4a9798ea3f97d3198 because the tablet was not found. No further retry: Not found: Tablet not found: 24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.195166 1417 ts_tablet_manager.cc:1918] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:36.195319 1417 log.cc:1196] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.195390 1181 tablet_service.cc:1514] Processing DeleteTablet for tablet 24862a0b81cb4da4a9798ea3f97d3198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:36.195228 428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 19:26:36.195760 1417 ts_tablet_manager.cc:1939] T 24862a0b81cb4da4a9798ea3f97d3198 P 732e68b58e084760840cd645eaa957e4: Deleting consensus metadata
I20260810 19:26:36.196202 1460 ts_tablet_manager.cc:1905] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 19:26:36.196233 428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:26:36.196696 403 catalog_manager.cc:4841] TS 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813): tablet 24862a0b81cb4da4a9798ea3f97d3198 (table test-table [id=860970ae84bd4e178a83c2af640e71c0]) successfully deleted
W20260810 19:26:36.197206 403 catalog_manager.cc:4808] TS 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813): delete failed for tablet 24862a0b81cb4da4a9798ea3f97d3198 because the tablet was not found. No further retry: Not found: Tablet not found: 24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.203925 676 tablet_service.cc:1467] Processing CreateTablet for tablet 2008565868034e6cac182fe7d1502b22 (DEFAULT_TABLE table=test-table [id=a0b248ee420a47768716ec6f7dd91625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.204381 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2008565868034e6cac182fe7d1502b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.205457 1181 tablet_service.cc:1467] Processing CreateTablet for tablet 2008565868034e6cac182fe7d1502b22 (DEFAULT_TABLE table=test-table [id=a0b248ee420a47768716ec6f7dd91625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.205802 1181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2008565868034e6cac182fe7d1502b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.209954 926 tablet_service.cc:1467] Processing CreateTablet for tablet 2008565868034e6cac182fe7d1502b22 (DEFAULT_TABLE table=test-table [id=a0b248ee420a47768716ec6f7dd91625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:26:36.210345 1460 ts_tablet_manager.cc:1918] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:26:36.210448 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2008565868034e6cac182fe7d1502b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:36.210580 1359 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Bootstrap starting.
I20260810 19:26:36.210626 1460 log.cc:1196] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/24862a0b81cb4da4a9798ea3f97d3198
I20260810 19:26:36.211192 1460 ts_tablet_manager.cc:1939] T 24862a0b81cb4da4a9798ea3f97d3198 P 328810c709c84794889c6b3d12221d27: Deleting consensus metadata
I20260810 19:26:36.212400 407 catalog_manager.cc:4841] TS 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617): tablet 24862a0b81cb4da4a9798ea3f97d3198 (table test-table [id=860970ae84bd4e178a83c2af640e71c0]) successfully deleted
I20260810 19:26:36.212865 1359 tablet_bootstrap.cc:654] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.213421 1362 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Bootstrap starting.
I20260810 19:26:36.214968 1362 tablet_bootstrap.cc:654] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.218549 1361 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Bootstrap starting.
I20260810 19:26:36.220359 1362 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: No bootstrap required, opened a new log
I20260810 19:26:36.220537 1362 ts_tablet_manager.cc:1397] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 19:26:36.220623 1361 tablet_bootstrap.cc:654] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:36.221205 1359 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: No bootstrap required, opened a new log
I20260810 19:26:36.221391 1359 ts_tablet_manager.cc:1397] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260810 19:26:36.221346 1362 raft_consensus.cc:348] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.221655 1362 raft_consensus.cc:374] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.221735 1362 raft_consensus.cc:729] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.221968 1362 consensus_queue.cc:260] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.222147 1359 raft_consensus.cc:348] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.222407 1359 raft_consensus.cc:374] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.222482 1362 ts_tablet_manager.cc:1428] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:26:36.222497 1359 raft_consensus.cc:729] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328810c709c84794889c6b3d12221d27, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.222764 1359 consensus_queue.cc:260] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.223681 1361 tablet_bootstrap.cc:492] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: No bootstrap required, opened a new log
I20260810 19:26:36.223922 1361 ts_tablet_manager.cc:1397] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 19:26:36.224781 1361 raft_consensus.cc:348] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.225135 1361 raft_consensus.cc:374] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:36.225292 1361 raft_consensus.cc:729] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b40f672c95246c0bb233d4f96315301, State: Initialized, Role: FOLLOWER
I20260810 19:26:36.225720 1359 ts_tablet_manager.cc:1428] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260810 19:26:36.225641 1361 consensus_queue.cc:260] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.226380 1361 ts_tablet_manager.cc:1428] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:26:36.430326 1383 raft_consensus.cc:482] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:26:36.430557 1383 raft_consensus.cc:504] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.432379 1383 leader_election.cc:288] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847)
I20260810 19:26:36.441123 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.434310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:51339 (local address 127.19.68.2:46847)
0810 19:26:36.434464 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.434470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.434755 (+ 285us) server_negotiation.cc:407] Connection header received
0810 19:26:36.435081 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.435086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.435173 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.435265 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:36.435891 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.436785 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.438798 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.440060 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.440207 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.440315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.440641 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.440786 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:26:36.441176 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.434123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46847 (local address 127.19.68.3:51339)
0810 19:26:36.434650 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:36.434668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:36.434867 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:36.435290 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:36.435295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:36.435322 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:36.435696 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.435708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.436926 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.436930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.438645 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.438658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.440272 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.440278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.440463 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.440503 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:26:36.440602 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:26:36.440744 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":2,"thread_start_us":356,"threads_started":1}
I20260810 19:26:36.441352 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.432602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39617 (local address 127.19.68.3:40295)
0810 19:26:36.433928 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:36.433948 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:36.434104 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:36.434783 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:36.434788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:36.434818 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:26:36.435141 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.435154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.436544 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.436549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.438821 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.438838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.440500 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.440505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.440698 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.440731 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:26:36.440816 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 19:26:36.440943 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"spinlock_wait_cycles":14592,"thread_start_us":1051,"threads_started":1}
I20260810 19:26:36.441885 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.432781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:40295 (local address 127.19.68.1:39617)
0810 19:26:36.433732 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.433738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.434035 (+ 297us) server_negotiation.cc:407] Connection header received
0810 19:26:36.434561 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.434566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.434650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.434743 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:26:36.435361 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.436397 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.438974 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.440190 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.440296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.440421 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.441436 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.441571 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":623,"threads_started":1}
I20260810 19:26:36.442412 969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2008565868034e6cac182fe7d1502b22" candidate_uuid: "732e68b58e084760840cd645eaa957e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b40f672c95246c0bb233d4f96315301" is_pre_election: true
I20260810 19:26:36.442544 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2008565868034e6cac182fe7d1502b22" candidate_uuid: "732e68b58e084760840cd645eaa957e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27" is_pre_election: true
I20260810 19:26:36.442818 721 raft_consensus.cc:2454] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732e68b58e084760840cd645eaa957e4 in term 0.
I20260810 19:26:36.442838 969 raft_consensus.cc:2454] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732e68b58e084760840cd645eaa957e4 in term 0.
I20260810 19:26:36.443367 1081 leader_election.cc:302] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b40f672c95246c0bb233d4f96315301, 732e68b58e084760840cd645eaa957e4; no voters:
I20260810 19:26:36.443768 1383 raft_consensus.cc:2781] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:26:36.443903 1383 raft_consensus.cc:482] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:26:36.444010 1383 raft_consensus.cc:3037] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.447132 1383 raft_consensus.cc:504] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.447784 1383 leader_election.cc:288] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [CANDIDATE]: Term 1 election: Requested vote from peers 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617), 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847)
I20260810 19:26:36.448303 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2008565868034e6cac182fe7d1502b22" candidate_uuid: "732e68b58e084760840cd645eaa957e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328810c709c84794889c6b3d12221d27"
I20260810 19:26:36.448355 969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2008565868034e6cac182fe7d1502b22" candidate_uuid: "732e68b58e084760840cd645eaa957e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b40f672c95246c0bb233d4f96315301"
I20260810 19:26:36.448546 721 raft_consensus.cc:3037] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.448561 969 raft_consensus.cc:3037] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:36.451483 721 raft_consensus.cc:2454] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732e68b58e084760840cd645eaa957e4 in term 1.
I20260810 19:26:36.451881 1081 leader_election.cc:302] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328810c709c84794889c6b3d12221d27, 732e68b58e084760840cd645eaa957e4; no voters:
I20260810 19:26:36.451843 969 raft_consensus.cc:2454] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732e68b58e084760840cd645eaa957e4 in term 1.
I20260810 19:26:36.452463 1383 raft_consensus.cc:2781] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:36.452884 1383 raft_consensus.cc:686] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 1 LEADER]: Becoming Leader. State: Replica: 732e68b58e084760840cd645eaa957e4, State: Running, Role: LEADER
I20260810 19:26:36.453292 1383 consensus_queue.cc:237] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } }
I20260810 19:26:36.456933 429 catalog_manager.cc:5495] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732e68b58e084760840cd645eaa957e4 (127.19.68.3). New cstate: current_term: 1 leader_uuid: "732e68b58e084760840cd645eaa957e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732e68b58e084760840cd645eaa957e4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 } health_report { overall_health: UNKNOWN } } }
I20260810 19:26:36.527277 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.521288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.19.68.3:34813)
0810 19:26:36.521936 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.521941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.521961 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:26:36.522045 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.522048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.522124 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.522317 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:36.522714 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.523589 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.524871 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.525354 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.525456 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.526012 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.526638 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.526773 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.526920 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.526991 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":352,"threads_started":1}
I20260810 19:26:36.541807 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.536808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:53439 (local address 127.19.68.2:46847)
0810 19:26:36.537068 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.537073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.537372 (+ 299us) server_negotiation.cc:407] Connection header received
0810 19:26:36.537530 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.537534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.537611 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.537690 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:36.538125 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.538900 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.540208 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.540671 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.540777 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.541181 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.541294 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.541381 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.541481 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.541548 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 19:26:36.541899 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.536689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46847 (local address 127.19.68.3:53439)
0810 19:26:36.537258 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:36.537273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:36.537420 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:36.537707 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:36.537709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:36.537718 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:26:36.537996 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.538007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.539028 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.539032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.540086 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.540098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.540835 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.540839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.540958 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.541008 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.541386 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.541395 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:26:36.541452 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 19:26:36.541548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260810 19:26:36.550256 721 raft_consensus.cc:1261] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Refusing update from remote peer 732e68b58e084760840cd645eaa957e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:36.550596 969 raft_consensus.cc:1261] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Refusing update from remote peer 732e68b58e084760840cd645eaa957e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:26:36.550940 1507 consensus_queue.cc:1035] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328810c709c84794889c6b3d12221d27" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:36.551339 1383 consensus_queue.cc:1035] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b40f672c95246c0bb233d4f96315301" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:26:36.557580 1395 mvcc.cc:205] Tried to move back new op lower bound from 7317053425864777728 to 7317053425476829184. Current Snapshot: MvccSnapshot[applied={T|T < 7317053425864777728}]
I20260810 19:26:36.578990 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.573943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:34807 (local address 127.19.68.3:34813)
0810 19:26:36.574097 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.574101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.574507 (+ 406us) server_negotiation.cc:407] Connection header received
0810 19:26:36.574632 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.574635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.574695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.574762 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:26:36.575110 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.575838 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.577588 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.577997 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.578114 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.578458 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.578548 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.578608 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.578694 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.578751 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:26:36.579026 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.573916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:34813 (local address 127.19.68.2:34807)
0810 19:26:36.574429 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:36.574445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:36.574548 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:36.574767 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:36.574770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:36.574778 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:26:36.575013 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.575020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.575995 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.576000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.577377 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:36.577392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.578149 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.578153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:36.578282 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.578345 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:26:36.578615 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:26:36.578622 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:26:36.578673 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 19:26:36.578767 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260810 19:26:36.609972 429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:36.610261 429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:26:36.613310 429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28908ffa30b9442f8da979a17a30406e: Sending DeleteTablet for 3 replicas of tablet 2008565868034e6cac182fe7d1502b22
I20260810 19:26:36.614063 1181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2008565868034e6cac182fe7d1502b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49116
I20260810 19:26:36.614415 926 tablet_service.cc:1514] Processing DeleteTablet for tablet 2008565868034e6cac182fe7d1502b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55992
I20260810 19:26:36.614408 676 tablet_service.cc:1514] Processing DeleteTablet for tablet 2008565868034e6cac182fe7d1502b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:26:36 UTC) from {username='oryx', principal='oryx/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:56404
I20260810 19:26:36.614670 1415 tablet_replica.cc:330] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: stopping tablet replica
I20260810 19:26:36.614707 1417 tablet_replica.cc:330] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: stopping tablet replica
I20260810 19:26:36.614863 1415 raft_consensus.cc:2229] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:36.614918 1417 raft_consensus.cc:2229] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:26:36.615062 1415 raft_consensus.cc:2258] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.615226 1460 tablet_replica.cc:330] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: stopping tablet replica
I20260810 19:26:36.615424 1460 raft_consensus.cc:2229] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:26:36.615636 1460 raft_consensus.cc:2258] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.616087 1415 ts_tablet_manager.cc:1905] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:36.616640 1417 raft_consensus.cc:2258] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:26:36.617537 1460 ts_tablet_manager.cc:1905] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:36.618113 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 555
I20260810 19:26:36.620889 1417 ts_tablet_manager.cc:1905] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:26:36.624050 1460 ts_tablet_manager.cc:1918] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:36.624177 1460 log.cc:1196] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/2008565868034e6cac182fe7d1502b22
I20260810 19:26:36.624596 1460 ts_tablet_manager.cc:1939] T 2008565868034e6cac182fe7d1502b22 P 328810c709c84794889c6b3d12221d27: Deleting consensus metadata
I20260810 19:26:36.625694 407 catalog_manager.cc:4841] TS 328810c709c84794889c6b3d12221d27 (127.19.68.1:39617): tablet 2008565868034e6cac182fe7d1502b22 (table test-table [id=a0b248ee420a47768716ec6f7dd91625]) successfully deleted
I20260810 19:26:36.626339 1415 ts_tablet_manager.cc:1918] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:36.626464 1415 log.cc:1196] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/2008565868034e6cac182fe7d1502b22
I20260810 19:26:36.626798 1415 ts_tablet_manager.cc:1939] T 2008565868034e6cac182fe7d1502b22 P 5b40f672c95246c0bb233d4f96315301: Deleting consensus metadata
I20260810 19:26:36.627696 407 catalog_manager.cc:4841] TS 5b40f672c95246c0bb233d4f96315301 (127.19.68.2:46847): tablet 2008565868034e6cac182fe7d1502b22 (table test-table [id=a0b248ee420a47768716ec6f7dd91625]) successfully deleted
I20260810 19:26:36.631814 1417 ts_tablet_manager.cc:1918] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:26:36.631990 1417 log.cc:1196] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/2008565868034e6cac182fe7d1502b22
I20260810 19:26:36.632481 1417 ts_tablet_manager.cc:1939] T 2008565868034e6cac182fe7d1502b22 P 732e68b58e084760840cd645eaa957e4: Deleting consensus metadata
I20260810 19:26:36.633594 403 catalog_manager.cc:4841] TS 732e68b58e084760840cd645eaa957e4 (127.19.68.3:34813): tablet 2008565868034e6cac182fe7d1502b22 (table test-table [id=a0b248ee420a47768716ec6f7dd91625]) successfully deleted
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.1@KRBTEST.COM
I20260810 19:26:36.642583 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.619508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.19.68.1:39617)
0810 19:26:36.619882 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.619888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.625578 (+ 5690us) server_negotiation.cc:407] Connection header received
0810 19:26:36.625743 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.625748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.625828 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.625927 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:36.627377 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.628400 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.630256 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.630990 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.631141 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.636282 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:36.636300 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:36.636317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:36.636353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:36.639488 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.640200 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.640211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.640213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.640349 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.640758 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.640762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.640764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.641285 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:36.641627 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.641958 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.642217 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260810 19:26:36.643342 794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:36.739606 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 555
I20260810 19:26:36.769317 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 809
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.2@KRBTEST.COM
I20260810 19:26:36.784325 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.770314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.19.68.2:46847)
0810 19:26:36.770529 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.770533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.770549 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:26:36.770603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.770607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.770667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.770749 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:36.771941 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.772624 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.773908 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.774323 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.774442 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:36.778630 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:36.778643 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:36.778658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:36.778699 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:36.781593 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.782410 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.782417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.782419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.782556 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:36.782989 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:36.782994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:36.782996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:36.783317 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:36.783572 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:36.783915 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 19:26:36.784072 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:26:36.784934 1043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:36.947072 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 809
I20260810 19:26:36.992202 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 1057
Aug 10 19:26:36 dist-test-slave-2j7r krb5kdc[341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.68.3@KRBTEST.COM
I20260810 19:26:37.008297 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:36.993059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.19.68.3:34813)
0810 19:26:36.993287 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:36.993292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:36.993326 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:26:36.993547 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:36.993553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:36.993635 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:36.993751 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:36.995239 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.996017 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.997299 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:36.997792 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:36.997911 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:37.002177 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:37.002191 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:37.002207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:37.002243 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:37.005463 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:37.006175 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:37.006183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:37.006185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:37.006342 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:37.006725 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:37.006728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:37.006730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:37.007109 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:37.007443 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:37.007733 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 19:26:37.007975 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:26:37.008989 1304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:37.146643 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 1057
I20260810 19:26:37.178570 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 367
I20260810 19:26:37.179782 500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:37.299284 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 367
2026-08-10T19:26:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6015 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:37 dist-test-slave-2j7r krb5kdc[1551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:37 dist-test-slave-2j7r krb5kdc[1551](info): set up 2 sockets
Aug 10 19:26:37 dist-test-slave-2j7r krb5kdc[1551](info): commencing operation
krb5kdc: starting...
W20260810 19:26:38.547659 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.003s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:38 dist-test-slave-2j7r krb5kdc[1551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:26:38.813870 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40605
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:39.141492 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:39.141790 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:39.141927 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:39.149042 1577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:39.149170 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:39.149251 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:39.149326 1577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:39.149397 1577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:39.159130 1577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:40605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:39.160425 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:39.161800 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:39.173035 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:39.173079 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:39.173117 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:39.173815 1577 server_base.cc:1034] running on GCE node
I20260810 19:26:39.174569 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:39.175918 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:39.177181 1577 hybrid_clock.cc:648] HybridClock initialized: now 1786389999177113 us; error 38 us; skew 500 ppm
I20260810 19:26:39.177510 1577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T19:26:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1940 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:26:39 dist-test-slave-2j7r krb5kdc[1605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:26:39 dist-test-slave-2j7r krb5kdc[1605](info): set up 2 sockets
Aug 10 19:26:39 dist-test-slave-2j7r krb5kdc[1605](info): commencing operation
krb5kdc: starting...
W20260810 19:26:40.494956 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:26:40 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:26:40.750705 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39225
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:41.077251 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:41.077551 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:41.077641 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:41.084664 1632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:41.084770 1632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:41.084817 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:41.084874 1632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:41.084916 1632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:41.094810 1632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:41.096123 1632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:41.097460 1632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:41.108964 1642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:41.109007 1645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:41.109052 1643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:41.110070 1632 server_base.cc:1034] running on GCE node
I20260810 19:26:41.110828 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:41.112270 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:41.113476 1632 hybrid_clock.cc:648] HybridClock initialized: now 1786390001113418 us; error 43 us; skew 500 ppm
I20260810 19:26:41.113793 1632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:41 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390001, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:41.127264 1632 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:26:41.129031 1632 webserver.cc:460] Webserver started at http://127.19.68.62:34681/ using document root <none> and password file <none>
I20260810 19:26:41.129789 1632 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:41.129930 1632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:41.130267 1632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:41.133853 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "727443ba1c43435ca3b0b9a0b207d27f"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "ce1b2b1b4ba1269e94baebc2f1f83e01"
server_key_iv: "60df1030c87534fce0e11517682874eb"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.134617 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "727443ba1c43435ca3b0b9a0b207d27f"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "ce1b2b1b4ba1269e94baebc2f1f83e01"
server_key_iv: "60df1030c87534fce0e11517682874eb"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.139776 1632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:41.143051 1655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:41.144331 1632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:41.144506 1632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "727443ba1c43435ca3b0b9a0b207d27f"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "ce1b2b1b4ba1269e94baebc2f1f83e01"
server_key_iv: "60df1030c87534fce0e11517682874eb"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.144695 1632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:41.157263 1632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:41.161058 1632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:41.161360 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:41.179008 1632 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39225
I20260810 19:26:41.179044 1754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39225 every 8 connection(s)
I20260810 19:26:41.180379 1632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:41.182529 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 1632
I20260810 19:26:41.182760 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:26:41.183619 19728 external_mini_cluster.cc:1428] Setting key e4310131618b0cb4be90c1e8dbd2142b
I20260810 19:26:41.184816 1755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:26:41.196739 1755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Bootstrap starting.
Aug 10 19:26:41 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:41.200090 1755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Neither blocks nor log segments found. Creating new log.
I20260810 19:26:41.201210 1755 log.cc:824] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:41.204420 1755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: No bootstrap required, opened a new log
I20260810 19:26:41.206388 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:41.185799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.19.68.62:39225)
0810 19:26:41.186565 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:41.186576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:41.186618 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:26:41.188224 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:41.188245 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:41.188638 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:41.189060 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:41.190577 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.191907 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.193143 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.193858 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.194031 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:41.198293 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:41.198302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:41.198315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:41.198359 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:41.201709 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:41.202514 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:41.202523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:41.202529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:41.202711 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:41.203134 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:41.203137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:41.203139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:41.203711 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:41.203947 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:41.204475 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 19:26:41.204910 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260810 19:26:41.212519 1755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:41.212862 1755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:26:41.212965 1755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727443ba1c43435ca3b0b9a0b207d27f, State: Initialized, Role: FOLLOWER
I20260810 19:26:41.213599 1755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:41.213866 1755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:41.214008 1755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:41.214178 1755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:26:41.216274 1755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:41.216832 1755 leader_election.cc:302] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727443ba1c43435ca3b0b9a0b207d27f; no voters:
I20260810 19:26:41.217520 1755 leader_election.cc:288] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:26:41.217625 1774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:26:41.218248 1774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 LEADER]: Becoming Leader. State: Replica: 727443ba1c43435ca3b0b9a0b207d27f, State: Running, Role: LEADER
I20260810 19:26:41.218858 1774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:41.219754 1755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:41.221710 1776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727443ba1c43435ca3b0b9a0b207d27f. Latest consensus state: current_term: 1 leader_uuid: "727443ba1c43435ca3b0b9a0b207d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } } }
I20260810 19:26:41.222080 1776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:41.222591 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727443ba1c43435ca3b0b9a0b207d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } } }
I20260810 19:26:41.222905 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:41.223572 1781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:41.228960 1781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:41.238067 1781 catalog_manager.cc:1347] Generated new cluster ID: 24b19f6115f04359b63040ccf9394ce4
I20260810 19:26:41.238173 1781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:41.250907 1781 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:26:41.252195 1781 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:41.258155 1781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Generated new TSK 0
I20260810 19:26:41.258935 1781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:26:41.332399 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:41.664702 1805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:41.664983 1805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:41.665156 1805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:41.672201 1805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:41.672313 1805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:41.672467 1805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:41.682034 1805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:41.683305 1805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:41.684567 1805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:41.697218 1814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:41.697247 1819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:41.697749 1812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:41.698107 1805 server_base.cc:1034] running on GCE node
I20260810 19:26:41.698666 1805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:41.699563 1805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:41.700788 1805 hybrid_clock.cc:648] HybridClock initialized: now 1786390001700704 us; error 54 us; skew 500 ppm
I20260810 19:26:41.701103 1805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:41 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390001, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:41.704205 1805 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:26:41.705744 1805 webserver.cc:460] Webserver started at http://127.19.68.1:41347/ using document root <none> and password file <none>
I20260810 19:26:41.706430 1805 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:41.706547 1805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:41.706864 1805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:41.709820 1805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "c7d2fc32d40b45ae9f51e7bb8d597d3d"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "5ee5b204d38da4a585ea0bef2e5d4996"
server_key_iv: "0a06531dbf02862dc041495ba222c0b3"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.710565 1805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "c7d2fc32d40b45ae9f51e7bb8d597d3d"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "5ee5b204d38da4a585ea0bef2e5d4996"
server_key_iv: "0a06531dbf02862dc041495ba222c0b3"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.715431 1805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:26:41.718726 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:41.720006 1805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:26:41.720157 1805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "c7d2fc32d40b45ae9f51e7bb8d597d3d"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "5ee5b204d38da4a585ea0bef2e5d4996"
server_key_iv: "0a06531dbf02862dc041495ba222c0b3"
server_key_version: "encryptionkey@0"
I20260810 19:26:41.720341 1805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:41.733021 1805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:41.736593 1805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:41.736857 1805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:41.737818 1805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:41.739131 1805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:41.739209 1805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:41.739324 1805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:41.739383 1805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:41.770139 1805 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:37131
I20260810 19:26:41.770187 2034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:37131 every 8 connection(s)
I20260810 19:26:41.771373 1805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:41.779312 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 1805
I20260810 19:26:41.779503 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:26:41.780133 19728 external_mini_cluster.cc:1428] Setting key 74cf982ef9a78e8fafc021c5047763bc
Aug 10 19:26:41 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390001, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:41.791574 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:41.773796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51559 (local address 127.19.68.62:39225)
0810 19:26:41.774244 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:41.774248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:41.775065 (+ 817us) server_negotiation.cc:407] Connection header received
0810 19:26:41.776880 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:41.776886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:41.776968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:41.777066 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:41.778571 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.779285 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.780917 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.781494 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.781614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:41.785234 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:41.785240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:41.785244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:41.785283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:41.788188 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:41.789044 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:41.789049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:41.789052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:41.789218 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:41.789697 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:41.789701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:41.789704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:41.790038 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:41.790176 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:41.791014 (+ 838us) server_negotiation.cc:299] Negotiation successful
0810 19:26:41.791238 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260810 19:26:41.792582 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:41.774180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:51559)
0810 19:26:41.774894 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:41.774943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:41.776604 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:41.777345 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:41.777354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:41.777850 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:41.778390 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:41.778408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.779451 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.779457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:41.780763 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:41.780777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:41.781727 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:41.781734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:41.781977 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:41.782610 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:41.782631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:41.784992 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:41.788354 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:41.788364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:41.788379 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:41.788916 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:41.789325 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:41.789330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:41.789333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:41.789578 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:41.790178 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:41.790185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:41.790592 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 19:26:41.790892 (+ 300us) client_negotiation.cc:240] Negotiation successful
0810 19:26:41.791377 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":266,"threads_started":1}
I20260810 19:26:41.794282 2035 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39225
I20260810 19:26:41.794580 2035 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:41.795298 2035 heartbeater.cc:511] Master 127.19.68.62:39225 requested a full tablet report, sending...
I20260810 19:26:41.797366 1680 ts_manager.cc:194] Registered new tserver with Master: c7d2fc32d40b45ae9f51e7bb8d597d3d (127.19.68.1:37131)
I20260810 19:26:41.798832 1680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:51559
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:26:41.840961 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:42.165802 2043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:42.166131 2043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:42.166285 2043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:42.173377 2043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:42.173472 2043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:42.173611 2043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:42.183187 2043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:42.184412 2043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:42.185731 2043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:42.198760 2048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:42.198669 2070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:42.198704 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:42.199745 2043 server_base.cc:1034] running on GCE node
I20260810 19:26:42.200395 2043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:42.201436 2043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:42.202663 2043 hybrid_clock.cc:648] HybridClock initialized: now 1786390002202605 us; error 44 us; skew 500 ppm
I20260810 19:26:42.202980 2043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:42 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390002, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:42.206267 2043 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:26:42.207736 2043 webserver.cc:460] Webserver started at http://127.19.68.2:33491/ using document root <none> and password file <none>
I20260810 19:26:42.208431 2043 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:42.208552 2043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:42.208901 2043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:42.211863 2043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "36ca74ea52a6469e9d0b0fea3a532ef5"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "546ad24feea27ab3703f0d17a814c9cb"
server_key_iv: "6e44552312dc780351729e53c40898dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.212657 2043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "36ca74ea52a6469e9d0b0fea3a532ef5"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "546ad24feea27ab3703f0d17a814c9cb"
server_key_iv: "6e44552312dc780351729e53c40898dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.218070 2043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 19:26:42.221545 2082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.222841 2043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:26:42.223039 2043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "36ca74ea52a6469e9d0b0fea3a532ef5"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "546ad24feea27ab3703f0d17a814c9cb"
server_key_iv: "6e44552312dc780351729e53c40898dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.223224 2043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:42.255962 2043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:42.260167 2043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:42.260447 2043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:42.261435 2043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:42.263002 2043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:42.263095 2043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.263227 2043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:42.263279 2043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.292791 2043 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:42079
I20260810 19:26:42.292861 2291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:42079 every 8 connection(s)
I20260810 19:26:42.294113 2043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:42.297362 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 2043
I20260810 19:26:42.297549 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:26:42.298143 19728 external_mini_cluster.cc:1428] Setting key 7e40f865c48850995a15273d823ee3e1
Aug 10 19:26:42 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390002, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:42.315972 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:42.296812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:57281 (local address 127.19.68.62:39225)
0810 19:26:42.297341 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:42.297345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:42.300457 (+ 3112us) server_negotiation.cc:407] Connection header received
0810 19:26:42.300517 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:42.300521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:42.300586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:42.300690 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:42.302320 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.303079 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.304702 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.305216 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.305348 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:42.309493 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:42.309499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:42.309503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:42.309539 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:42.312749 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:42.313531 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:42.313537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:42.313541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:42.313731 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:42.314152 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:42.314158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:42.314161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:42.314528 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:42.314686 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:42.315326 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 19:26:42.315551 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260810 19:26:42.316596 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:42.297322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:57281)
0810 19:26:42.298107 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:42.298147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:42.300133 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:42.300970 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:42.300979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:42.301550 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:42.302124 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:42.302143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.303251 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.303256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:42.304540 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:42.304554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.305456 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.305465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:42.305710 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:42.306773 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:42.306806 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:42.309232 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:42.312917 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:42.312924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:42.312935 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:42.313389 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:42.313835 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:42.313838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:42.313840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:42.314032 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:42.314639 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:42.314644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:42.314952 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 19:26:42.315168 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 19:26:42.315537 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":328,"threads_started":1}
I20260810 19:26:42.318094 2295 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39225
I20260810 19:26:42.318449 2295 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:42.319336 2295 heartbeater.cc:511] Master 127.19.68.62:39225 requested a full tablet report, sending...
I20260810 19:26:42.321112 1680 ts_manager.cc:194] Registered new tserver with Master: 36ca74ea52a6469e9d0b0fea3a532ef5 (127.19.68.2:42079)
I20260810 19:26:42.321910 1680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:57281
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:26:42.360543 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:42.699940 2305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:42.700248 2305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:42.700418 2305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:42.708424 2305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:42.708534 2305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:42.708699 2305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:42.718688 2305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:42.719974 2305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:42.721488 2305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:42.735352 2310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:42.735419 2311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:42.735391 2314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:42.735976 2305 server_base.cc:1034] running on GCE node
I20260810 19:26:42.736583 2305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:42.737560 2305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:42.738783 2305 hybrid_clock.cc:648] HybridClock initialized: now 1786390002738722 us; error 37 us; skew 500 ppm
I20260810 19:26:42.739084 2305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:42 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390002, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:42.742341 2305 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:26:42.743759 2305 webserver.cc:460] Webserver started at http://127.19.68.3:43317/ using document root <none> and password file <none>
I20260810 19:26:42.744390 2305 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:42.744486 2305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:42.744738 2305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:26:42.747612 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "82b2c7b146a8434bba7d3f9550628bb6"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "f67d27b1f75b2f9586614162c1b4667a"
server_key_iv: "091dbf774e05fc929ac3620da24d3c76"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.748390 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "82b2c7b146a8434bba7d3f9550628bb6"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "f67d27b1f75b2f9586614162c1b4667a"
server_key_iv: "091dbf774e05fc929ac3620da24d3c76"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.753415 2305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 19:26:42.756620 2326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.757929 2305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:26:42.758075 2305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "82b2c7b146a8434bba7d3f9550628bb6"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "f67d27b1f75b2f9586614162c1b4667a"
server_key_iv: "091dbf774e05fc929ac3620da24d3c76"
server_key_version: "encryptionkey@0"
I20260810 19:26:42.758257 2305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:42.784403 2305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:42.788327 2305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:42.788578 2305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:42.789489 2305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:42.790758 2305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:42.790830 2305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.790937 2305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:42.790992 2305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:42.801775 2035 heartbeater.cc:503] Master 127.19.68.62:39225 was elected leader, sending a full tablet report...
I20260810 19:26:42.821642 2305 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:43755
I20260810 19:26:42.821707 2515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:43755 every 8 connection(s)
I20260810 19:26:42.822921 2305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:42.825650 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 2305
I20260810 19:26:42.825821 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:26:42.826396 19728 external_mini_cluster.cc:1428] Setting key dc570d9bdd7105bfac4b6b48eb9e4c50
Aug 10 19:26:42 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390002, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:26:42.843456 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:42.825509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:47175 (local address 127.19.68.62:39225)
0810 19:26:42.825941 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:42.825946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:42.827371 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 19:26:42.829876 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:42.829881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:42.829969 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:42.830058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:42.831526 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.832284 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.833862 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.834257 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.834367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:42.837822 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:42.837827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:42.837830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:42.837862 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:42.840336 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:42.841044 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:42.841048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:42.841050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:42.841221 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:42.841659 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:42.841663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:42.841666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:42.841939 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:42.842080 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:42.842820 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 19:26:42.843090 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":196,"threads_started":1}
I20260810 19:26:42.844273 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:42.826223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:47175)
0810 19:26:42.827206 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:42.827271 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:42.829623 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:42.830300 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:42.830308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:42.830823 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:42.831365 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:42.831383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.832434 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.832439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:42.833714 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:42.833739 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:42.834471 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:42.834479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:42.834856 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:42.835483 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:42.835504 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:42.837627 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:26:42.840476 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:42.840485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:42.840500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:42.840911 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:42.841330 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:26:42.841336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:26:42.841339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:26:42.841537 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:26:42.842070 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:26:42.842078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:26:42.842384 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 19:26:42.842628 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 19:26:42.843146 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":352,"threads_started":1}
I20260810 19:26:42.845506 2518 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39225
I20260810 19:26:42.845803 2518 heartbeater.cc:464] Registering TS with master...
I20260810 19:26:42.846517 2518 heartbeater.cc:511] Master 127.19.68.62:39225 requested a full tablet report, sending...
I20260810 19:26:42.847889 1680 ts_manager.cc:194] Registered new tserver with Master: 82b2c7b146a8434bba7d3f9550628bb6 (127.19.68.3:43755)
I20260810 19:26:42.848701 1680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:47175
I20260810 19:26:42.853639 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
Entry for principal oryx/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab.
WARNING: no policy specified for oryx/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
Entry for principal oryx/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab.
WARNING: no policy specified for oryx/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
Entry for principal oryx/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab.
I20260810 19:26:43.017544 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 1805
Aug 10 19:26:43 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:26:43.034042 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:43.018848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.19.68.1:37131)
0810 19:26:43.019507 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:43.019514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:43.019544 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:26:43.019605 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:43.019612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:43.019822 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:43.019961 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:43.021593 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.022617 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.024029 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.024710 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.024825 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:43.028947 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:43.028958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:43.028970 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:43.029007 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:43.031537 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.032165 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.032171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.032174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.032295 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.032579 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.032581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.032583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.032913 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:43.033180 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:43.033461 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 19:26:43.033660 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":330,"threads_started":1}
I20260810 19:26:43.035382 2029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:43.118322 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 1805
I20260810 19:26:43.141760 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2043
Aug 10 19:26:43 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:26:43.157244 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:43.142699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.19.68.2:42079)
0810 19:26:43.143264 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:43.143271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:43.143295 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:26:43.143361 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:43.143367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:43.143561 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:43.143684 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:43.144958 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.146046 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.147277 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.147823 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.147920 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:43.152024 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:43.152035 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:43.152050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:43.152086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:43.154752 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.155338 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.155344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.155347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.155461 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.155852 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.155855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.155857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.156169 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:43.156391 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:43.156689 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 19:26:43.156880 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260810 19:26:43.158512 2279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:43.240145 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 2043
I20260810 19:26:43.266607 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2305
Aug 10 19:26:43 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:26:43.282475 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:43.267616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.19.68.3:43755)
0810 19:26:43.268196 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:43.268206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:43.268241 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:26:43.268309 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:43.268316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:43.268536 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:43.268670 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:43.270095 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.271159 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.272431 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:43.273004 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:43.273132 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:43.277315 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:43.277327 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:43.277341 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:43.277382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:43.279912 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.280503 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.280509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.280512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.280636 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:43.280956 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:43.280959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:43.280961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:43.281428 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:43.281667 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:43.281889 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 19:26:43.282109 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":253,"threads_started":1}
I20260810 19:26:43.283704 2508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:43.364938 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 2305
I20260810 19:26:43.389257 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 1632
I20260810 19:26:43.390522 1745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:26:43.494626 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 1632
I20260810 19:26:43.521617 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39225
--webserver_interface=127.19.68.62
--webserver_port=34681
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:26:43.849768 2550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:43.850072 2550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:43.850207 2550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:43.857463 2550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:26:43.857568 2550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:43.857640 2550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:43.857717 2550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:26:43.857779 2550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:26:43.867959 2550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:43.869378 2550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:43.870730 2550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:43.882344 2556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:43.882289 2561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:43.882378 2565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:43.882879 2550 server_base.cc:1034] running on GCE node
I20260810 19:26:43.883702 2550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:43.885113 2550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:43.886332 2550 hybrid_clock.cc:648] HybridClock initialized: now 1786390003886266 us; error 53 us; skew 500 ppm
I20260810 19:26:43.886632 2550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:43 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:43.889627 2550 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:26:43.891078 2550 webserver.cc:460] Webserver started at http://127.19.68.62:34681/ using document root <none> and password file <none>
I20260810 19:26:43.891709 2550 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:43.891832 2550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:43.897130 2550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:43.899859 2578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:43.901486 2550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:26:43.901654 2550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "727443ba1c43435ca3b0b9a0b207d27f"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "ce1b2b1b4ba1269e94baebc2f1f83e01"
server_key_iv: "60df1030c87534fce0e11517682874eb"
server_key_version: "encryptionkey@0"
I20260810 19:26:43.902626 2550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:43.924602 2550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:43.928741 2550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:43.928999 2550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:43.947067 2550 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39225
I20260810 19:26:43.947103 2672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39225 every 8 connection(s)
I20260810 19:26:43.948607 2550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:26:43.955330 2673 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:26:43.957243 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 2550
I20260810 19:26:43.958039 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:37131
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:39225
--webserver_port=41347
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:26:43.958568 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Bootstrap starting.
I20260810 19:26:43.979561 2673 log.cc:824] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:26:43.991317 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:26:43.991979 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Bootstrap complete.
I20260810 19:26:44.005921 2673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:44.007138 2673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727443ba1c43435ca3b0b9a0b207d27f, State: Initialized, Role: FOLLOWER
I20260810 19:26:44.007858 2673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:44.008131 2673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:26:44.008260 2673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:26:44.008435 2673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:26:44.013931 2673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:44.014528 2673 leader_election.cc:302] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727443ba1c43435ca3b0b9a0b207d27f; no voters:
I20260810 19:26:44.015167 2673 leader_election.cc:288] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:26:44.015347 2680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:26:44.016512 2680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [term 2 LEADER]: Becoming Leader. State: Replica: 727443ba1c43435ca3b0b9a0b207d27f, State: Running, Role: LEADER
I20260810 19:26:44.017139 2680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } }
I20260810 19:26:44.017344 2673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:26:44.019691 2682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727443ba1c43435ca3b0b9a0b207d27f. Latest consensus state: current_term: 2 leader_uuid: "727443ba1c43435ca3b0b9a0b207d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } } }
I20260810 19:26:44.020056 2682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:44.020588 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "727443ba1c43435ca3b0b9a0b207d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727443ba1c43435ca3b0b9a0b207d27f" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39225 } } }
I20260810 19:26:44.020933 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f [sys.catalog]: This master's current role is: LEADER
I20260810 19:26:44.030184 2690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:26:44.035802 2690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:26:44.037297 2690 catalog_manager.cc:1259] Loaded cluster ID: 24b19f6115f04359b63040ccf9394ce4
I20260810 19:26:44.037452 2690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:26:44.041517 2690 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:26:44.042997 2690 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 727443ba1c43435ca3b0b9a0b207d27f: Loaded TSK: 0
I20260810 19:26:44.044623 2690 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:26:44.317516 2675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:44.317824 2675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:44.317953 2675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:44.325380 2675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:44.325484 2675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:44.325639 2675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:26:44.335558 2675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.1.keytab
--principal=oryx/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:37131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=41347
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:44.336838 2675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:44.338145 2675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:44.350975 2732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:44.351029 2733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:44.351769 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:44.351862 2675 server_base.cc:1034] running on GCE node
I20260810 19:26:44.352478 2675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:44.353482 2675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:44.354702 2675 hybrid_clock.cc:648] HybridClock initialized: now 1786390004354645 us; error 50 us; skew 500 ppm
I20260810 19:26:44.355001 2675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390004, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:44.358096 2675 init.cc:377] Logged in from keytab as oryx/127.19.68.1@KRBTEST.COM (short username oryx)
I20260810 19:26:44.359644 2675 webserver.cc:460] Webserver started at http://127.19.68.1:41347/ using document root <none> and password file <none>
I20260810 19:26:44.360344 2675 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:44.360476 2675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:44.365926 2675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:26:44.368619 2748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.369961 2675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:26:44.370100 2675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "c7d2fc32d40b45ae9f51e7bb8d597d3d"
format_stamp: "Formatted at 2026-08-10 19:26:41 on dist-test-slave-2j7r"
server_key: "5ee5b204d38da4a585ea0bef2e5d4996"
server_key_iv: "0a06531dbf02862dc041495ba222c0b3"
server_key_version: "encryptionkey@0"
I20260810 19:26:44.371083 2675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:44.396801 2675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:44.400934 2675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:44.401258 2675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:44.402242 2675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:44.403625 2675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:44.403713 2675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.403827 2675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:44.403903 2675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.435647 2675 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:37131
I20260810 19:26:44.435727 2970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:37131 every 8 connection(s)
I20260810 19:26:44.437141 2675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:26:44.442148 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 2675
I20260810 19:26:44.442894 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:42079
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:39225
--webserver_port=33491
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:44.456053 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.440091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:44883)
0810 19:26:44.440880 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.440934 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.443197 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.444625 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.444635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.445325 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.446047 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.446072 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.447716 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.447721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.449411 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.449426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.450435 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.450442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.450686 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.451355 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.451384 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.455181 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":332,"threads_started":1}
W20260810 19:26:44.456640 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.440070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:44883 (local address 127.19.68.62:39225)
0810 19:26:44.440751 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.440763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.441059 (+ 296us) server_negotiation.cc:407] Connection header received
0810 19:26:44.443769 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.443783 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.444095 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.444387 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.446266 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.447547 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.449550 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.450187 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.450330 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.455612 (+ 5282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:44883: BlockingRecv error: recv got EOF from 127.19.68.1:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":268,"threads_started":1}
W20260810 19:26:44.456817 2973 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39225 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.68.62:39225: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:44.465183 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.457719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:36085)
0810 19:26:44.457936 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.457951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.458029 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.458403 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.458408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.458713 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.458838 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.458845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.459821 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.459826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.460690 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.460701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.461396 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.461402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.461557 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.462073 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.462091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.464882 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 19:26:44.465334 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.457967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:36085 (local address 127.19.68.62:39225)
0810 19:26:44.458163 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.458168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.458182 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:26:44.458218 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.458221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.458273 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.458354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.458974 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.459683 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.460833 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.461256 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.461371 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.465127 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:36085: BlockingRecv error: recv got EOF from 127.19.68.1:36085 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:44.474435 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.465828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:36445)
0810 19:26:44.466084 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.466102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.466204 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.466796 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.466801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.467159 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.467362 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.467373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.468335 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.468340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.469793 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.469809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.470478 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.470483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.470662 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.471192 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.471216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.474161 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":617,"threads_started":1}
W20260810 19:26:44.474828 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.466337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:36445 (local address 127.19.68.62:39225)
0810 19:26:44.466508 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.466513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.466532 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:26:44.466580 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.466584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.466666 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.466750 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.467488 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.468185 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.469936 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.470332 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.470424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.474462 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:36445: BlockingRecv error: recv got EOF from 127.19.68.1:36445 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:26:44.474790 2973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:26:44.777808 2977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:44.778101 2977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:44.778251 2977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:44.785375 2977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:44.785498 2977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:44.785665 2977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:26:44.795549 2977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.2.keytab
--principal=oryx/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:42079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=33491
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:44.796892 2977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:44.798300 2977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:44.811784 2991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:44.811784 2992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:44.811779 2998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:44.812882 2977 server_base.cc:1034] running on GCE node
I20260810 19:26:44.813570 2977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:44.814539 2977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:44.815798 2977 hybrid_clock.cc:648] HybridClock initialized: now 1786390004815710 us; error 56 us; skew 500 ppm
I20260810 19:26:44.816125 2977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390004, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:44.819274 2977 init.cc:377] Logged in from keytab as oryx/127.19.68.2@KRBTEST.COM (short username oryx)
I20260810 19:26:44.820618 2977 webserver.cc:460] Webserver started at http://127.19.68.2:33491/ using document root <none> and password file <none>
I20260810 19:26:44.821347 2977 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:44.821484 2977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:44.826695 2977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:26:44.829305 3014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.830545 2977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:26:44.830695 2977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "36ca74ea52a6469e9d0b0fea3a532ef5"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "546ad24feea27ab3703f0d17a814c9cb"
server_key_iv: "6e44552312dc780351729e53c40898dd"
server_key_version: "encryptionkey@0"
I20260810 19:26:44.831669 2977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:44.844761 2977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:44.848730 2977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:44.848994 2977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:44.850085 2977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:44.851888 2977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:44.852006 2977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.852128 2977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:44.852219 2977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:44.886883 2977 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:42079
I20260810 19:26:44.886962 3236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:42079 every 8 connection(s)
I20260810 19:26:44.888291 2977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:26:44.891358 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 2977
I20260810 19:26:44.892143 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:43755
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:39225
--webserver_port=43317
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:26:44.905225 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.890710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:40147 (local address 127.19.68.62:39225)
0810 19:26:44.890883 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.890887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.892092 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 19:26:44.894478 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.894482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.894554 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.894637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.896477 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.897249 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.899230 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.899651 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.899743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.904945 (+ 5202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:40147: BlockingRecv error: recv got EOF from 127.19.68.2:40147 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:26:44.905706 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.891105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:40147)
0810 19:26:44.891892 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.891947 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.894180 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.894891 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.894901 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.895564 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.896276 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.896298 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.897416 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.897421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.899086 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.899107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.899869 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.899875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.900132 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.900812 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.900841 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.904661 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
W20260810 19:26:44.906527 3240 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39225 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.68.62:39225: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:44.917608 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.907073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:59333)
0810 19:26:44.907409 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.907423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.907521 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.907801 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.907804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.908090 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.908219 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.908227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.909478 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.909481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.910338 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.910349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.911151 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.911155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.911268 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.911809 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.911827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.917336 (+ 5509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229}
W20260810 19:26:44.918515 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.907198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:59333 (local address 127.19.68.62:39225)
0810 19:26:44.907376 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.907381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.907493 (+ 112us) server_negotiation.cc:407] Connection header received
0810 19:26:44.907616 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.907620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.907689 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.907779 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.908367 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.909349 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.910472 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.911017 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.911165 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.917854 (+ 6689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:59333: BlockingRecv error: recv got EOF from 127.19.68.2:59333 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:44 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:44.932642 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:44.918514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:46159)
0810 19:26:44.918744 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:44.918763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:44.918859 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:44.925013 (+ 6154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:44.925018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:44.925414 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:44.925629 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.925641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.927049 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.927054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.927997 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:44.928009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.928880 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.928884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:44.929039 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.929623 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:44.929641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:44.932430 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 19:26:44.932869 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:44.924113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:46159 (local address 127.19.68.62:39225)
0810 19:26:44.924745 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:44.924749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:44.924764 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:26:44.924806 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:44.924810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:44.924867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:44.924956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:44.925784 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.926889 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.928132 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:44.928747 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:44.928866 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:44.932651 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:46159: BlockingRecv error: recv got EOF from 127.19.68.2:46159 (error 108)
Metrics: {"server-negotiator.queue_time_us":410}
W20260810 19:26:44.932961 3240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:26:45.228396 3251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:26:45.228680 3251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:26:45.228775 3251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:26:45.236169 3251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:26:45.236289 3251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:26:45.236416 3251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:26:45.246215 3251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/krb5kdc/oryx_127.19.68.3.keytab
--principal=oryx/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:43755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=43317
--tserver_master_addrs=127.19.68.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:26:45.247581 3251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:26:45.248811 3251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:26:45.261366 3267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:45.261341 3272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:26:45.261433 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:26:45.262624 3251 server_base.cc:1034] running on GCE node
I20260810 19:26:45.263252 3251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:26:45.264149 3251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:26:45.265370 3251 hybrid_clock.cc:648] HybridClock initialized: now 1786390005265322 us; error 32 us; skew 500 ppm
I20260810 19:26:45.265678 3251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390005, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:26:45.268795 3251 init.cc:377] Logged in from keytab as oryx/127.19.68.3@KRBTEST.COM (short username oryx)
I20260810 19:26:45.270310 3251 webserver.cc:460] Webserver started at http://127.19.68.3:43317/ using document root <none> and password file <none>
I20260810 19:26:45.270944 3251 fs_manager.cc:359] Metadata directory not provided
I20260810 19:26:45.271059 3251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:26:45.276368 3251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:26:45.278998 3283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:45.280385 3251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:26:45.280553 3251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "82b2c7b146a8434bba7d3f9550628bb6"
format_stamp: "Formatted at 2026-08-10 19:26:42 on dist-test-slave-2j7r"
server_key: "f67d27b1f75b2f9586614162c1b4667a"
server_key_iv: "091dbf774e05fc929ac3620da24d3c76"
server_key_version: "encryptionkey@0"
I20260810 19:26:45.281554 3251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:26:45.304347 3251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:26:45.308168 3251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:26:45.308429 3251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:26:45.309423 3251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:26:45.310747 3251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:26:45.310850 3251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:45.310966 3251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:26:45.311035 3251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:26:45.342164 3251 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:43755
I20260810 19:26:45.342229 3499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:43755 every 8 connection(s)
I20260810 19:26:45.343598 3251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:26:45.352996 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 3251
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:26:45.360181 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:45.346077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:57993 (local address 127.19.68.62:39225)
0810 19:26:45.346290 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.346297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.347580 (+ 1283us) server_negotiation.cc:407] Connection header received
0810 19:26:45.349326 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.349330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.349397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.349480 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.350841 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.351566 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.353263 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.353734 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.353883 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.359918 (+ 6035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:57993: BlockingRecv error: recv got EOF from 127.19.68.3:57993 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:26:45.360790 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.346575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:57993)
0810 19:26:45.347354 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:45.347429 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:45.349061 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:45.349665 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:45.349672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:45.350140 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:45.350689 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.350705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.351684 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.351687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.353129 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.353145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.354021 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.354030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.354325 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.355081 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:45.355108 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:45.359635 (+ 4527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
W20260810 19:26:45.361578 3502 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39225 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.68.62:39225: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:45.369851 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.354047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.19.68.62:39225)
0810 19:26:45.354816 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.354821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.354839 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:26:45.354884 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.354888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.354949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.355038 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.356506 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.357400 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.358896 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.359503 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.359619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.362853 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:26:45.362861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:26:45.362875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:26:45.362914 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:26:45.366393 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:45.367157 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:45.367166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:45.367173 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:45.367361 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:26:45.367785 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:26:45.367790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:26:45.367792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:26:45.368263 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:26:45.368507 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 19:26:45.369047 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 19:26:45.369505 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":544,"threads_started":1}
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:45.370771 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.362107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:49837)
0810 19:26:45.362287 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:45.362304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:45.362400 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:45.362673 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:45.362677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:45.362992 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:45.363138 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.363166 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.364587 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.364592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.365583 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.365595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.366538 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.366544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.366708 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.367561 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:45.367609 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:45.370577 (+ 2968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 19:26:45.371179 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:45.362251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:49837 (local address 127.19.68.62:39225)
0810 19:26:45.362380 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.362384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.362399 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:26:45.362495 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.362499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.362548 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.362618 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.363345 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.364437 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.365726 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.366395 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.366525 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.370799 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:49837: BlockingRecv error: recv got EOF from 127.19.68.3:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:45.379289 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.371374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:38649)
0810 19:26:45.371560 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:45.371580 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:45.371674 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:45.372504 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:45.372508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:45.372797 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:45.372976 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.372985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.374170 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.374174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.375172 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.375184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.375866 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.375870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.375995 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.376489 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:45.376505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:45.379083 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 19:26:45.379604 3502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:26:45.379616 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:45.371683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:38649 (local address 127.19.68.62:39225)
0810 19:26:45.371898 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.371903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.371919 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:26:45.371960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.371965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.372028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.372121 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.373131 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.374044 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.375323 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.375749 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.375842 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.379343 (+ 3501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:38649: BlockingRecv error: recv got EOF from 127.19.68.3:38649 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:45.483842 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.475435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:52417)
0810 19:26:45.475923 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:45.475940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:45.476056 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:45.476330 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:45.476333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:45.476783 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:45.476999 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.477009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.478019 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.478023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.478995 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.479008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.479851 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.479856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.480037 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.480794 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:45.480812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:45.483617 (+ 2805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
W20260810 19:26:45.484298 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:45.475499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:52417 (local address 127.19.68.62:39225)
0810 19:26:45.475663 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.475668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.476031 (+ 363us) server_negotiation.cc:407] Connection header received
0810 19:26:45.476138 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.476142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.476206 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.476308 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.477143 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.477895 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.479134 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.479659 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.479800 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.484019 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:52417: BlockingRecv error: recv got EOF from 127.19.68.1:52417 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:45 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:45.947995 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:45.933615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:48877)
0810 19:26:45.934137 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:45.934155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:45.934282 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:45.934691 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:45.934695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:45.935256 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:45.935473 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.935484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.936539 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.936543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.937614 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:45.937628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.938371 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.938376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:45.938496 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.943702 (+ 5206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:45.943722 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:45.947661 (+ 3939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
W20260810 19:26:45.948133 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:45.933709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:48877 (local address 127.19.68.62:39225)
0810 19:26:45.933874 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:45.933879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:45.934232 (+ 353us) server_negotiation.cc:407] Connection header received
0810 19:26:45.934391 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:45.934404 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:45.934486 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:45.934606 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:45.935621 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.936373 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.937753 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:45.938201 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:45.938294 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:45.947906 (+ 9612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:48877: BlockingRecv error: recv got EOF from 127.19.68.2:48877 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:46.393939 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:46.380196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:39751)
0810 19:26:46.380718 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:46.380738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:46.380875 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:46.381254 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:46.381259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:46.381730 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:46.381935 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.381945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.382995 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.382999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.384087 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.384101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.384980 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.384987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.385181 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.390843 (+ 5662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:46.390863 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:46.393709 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
W20260810 19:26:46.394160 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:46.380361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:39751 (local address 127.19.68.62:39225)
0810 19:26:46.380573 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:46.380579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:46.380822 (+ 243us) server_negotiation.cc:407] Connection header received
0810 19:26:46.381020 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:46.381024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:46.381108 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:46.381191 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:46.382085 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.382819 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.384230 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.384800 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.384922 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.393934 (+ 9012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:39751: BlockingRecv error: recv got EOF from 127.19.68.3:39751 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:46.493428 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:46.484832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:60941)
0810 19:26:46.485457 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:46.485478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:46.485649 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:46.485969 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:46.485972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:46.486413 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:46.486568 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.486577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.487772 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.487776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.488672 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.488683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.489376 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.489381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.489497 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.490075 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:46.490094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:46.493113 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":312,"threads_started":1}
W20260810 19:26:46.493793 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:46.484940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:60941 (local address 127.19.68.62:39225)
0810 19:26:46.485182 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:46.485187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:46.485619 (+ 432us) server_negotiation.cc:407] Connection header received
0810 19:26:46.485768 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:46.485771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:46.485843 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:46.485932 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:46.486705 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.487619 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.488801 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.489216 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.489313 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.493376 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:60941: BlockingRecv error: recv got EOF from 127.19.68.1:60941 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:46 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:46.957998 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:46.949148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:55755)
0810 19:26:46.949839 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:46.949856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:46.949991 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:46.950367 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:46.950372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:46.950886 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:46.951040 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.951050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.952233 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.952239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.953493 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:46.953509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.954196 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.954200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:46.954313 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.954855 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:46.954872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:46.957665 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":453,"threads_started":1}
W20260810 19:26:46.958153 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:46.949210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:55755 (local address 127.19.68.62:39225)
0810 19:26:46.949435 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:46.949443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:46.949947 (+ 504us) server_negotiation.cc:407] Connection header received
0810 19:26:46.950111 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:46.950116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:46.950187 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:46.950272 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:46.951194 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.952097 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.953623 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:46.954065 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:46.954161 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:46.957906 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:55755: BlockingRecv error: recv got EOF from 127.19.68.2:55755 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:47.404459 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:47.394935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:44237)
0810 19:26:47.395525 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:47.395548 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:47.395671 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:47.396009 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:47.396013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:47.396584 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:47.396832 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.396848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.398277 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.398281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.399649 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.399669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.400460 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.400465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.400592 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.401216 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:47.401236 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:47.404123 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
W20260810 19:26:47.404618 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:47.395129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:44237 (local address 127.19.68.62:39225)
0810 19:26:47.395308 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:47.395313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:47.395621 (+ 308us) server_negotiation.cc:407] Connection header received
0810 19:26:47.395766 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:47.395770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:47.395853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:47.395969 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:47.397016 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.398075 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.399822 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.400298 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.400393 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.404357 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:44237: BlockingRecv error: recv got EOF from 127.19.68.3:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:47.502468 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:47.494453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:45771)
0810 19:26:47.494953 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:47.494969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:47.495082 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:47.495367 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:47.495371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:47.495727 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:47.495904 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.495913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.496913 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.496917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.497937 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.497952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.498627 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.498632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.498797 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.499374 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:47.499393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:47.502205 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
W20260810 19:26:47.502632 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:47.494509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:45771 (local address 127.19.68.62:39225)
0810 19:26:47.494684 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:47.494688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:47.495033 (+ 345us) server_negotiation.cc:407] Connection header received
0810 19:26:47.495167 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:47.495170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:47.495230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:47.495308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:47.496016 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.496746 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.498069 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.498462 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.498554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.502412 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:45771: BlockingRecv error: recv got EOF from 127.19.68.1:45771 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:47 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:47.966645 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:47.958919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:57461)
0810 19:26:47.959363 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:47.959379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:47.959506 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:47.959793 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:47.959796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:47.960134 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:47.960306 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.960314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.961297 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.961301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.962264 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:47.962277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.962914 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.962918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:47.963045 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.963585 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:47.963601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:47.966374 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
W20260810 19:26:47.966796 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:47.958993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:57461 (local address 127.19.68.62:39225)
0810 19:26:47.959208 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:47.959214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:47.959452 (+ 238us) server_negotiation.cc:407] Connection header received
0810 19:26:47.959622 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:47.959625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:47.959687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:47.959755 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:47.960409 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.961188 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.962392 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:47.962803 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:47.962895 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:47.966596 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:57461: BlockingRecv error: recv got EOF from 127.19.68.2:57461 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:48.413812 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:48.405464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:56503)
0810 19:26:48.405974 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:48.405996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:48.406168 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:48.406520 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:48.406524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:48.406893 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:48.407060 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.407070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.408148 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.408154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.409048 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.409060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.409888 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.409894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.410067 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.410728 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:48.410752 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:48.413570 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
W20260810 19:26:48.413982 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:48.405575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:56503 (local address 127.19.68.62:39225)
0810 19:26:48.405780 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:48.405787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:48.406085 (+ 298us) server_negotiation.cc:407] Connection header received
0810 19:26:48.406292 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:48.406298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:48.406378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:48.406461 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:48.407207 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.408002 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.409216 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.409728 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.409835 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.413780 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:56503: BlockingRecv error: recv got EOF from 127.19.68.3:56503 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:48.512240 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:48.503486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:58843)
0810 19:26:48.504016 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:48.504033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:48.504170 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:48.504542 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:48.504546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:48.505037 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:48.505232 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.505284 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.506588 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.506592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.507604 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.507618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.508360 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.508365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.508509 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.509182 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:48.509200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:48.512008 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
W20260810 19:26:48.512470 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:48.503565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58843 (local address 127.19.68.62:39225)
0810 19:26:48.503849 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:48.503855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:48.504126 (+ 271us) server_negotiation.cc:407] Connection header received
0810 19:26:48.504297 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:48.504303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:48.504382 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:48.504488 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:48.505448 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.506457 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.507764 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.508190 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.508303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.512215 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:58843: BlockingRecv error: recv got EOF from 127.19.68.1:58843 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:48 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:48.975663 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:48.967546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:50737)
0810 19:26:48.968079 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:48.968097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:48.968214 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:48.968568 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:48.968571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:48.968930 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:48.969100 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.969110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.970080 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.970084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.971040 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:48.971052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.971698 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.971701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:48.971822 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.972417 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:48.972434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:48.975386 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
W20260810 19:26:48.975890 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:48.967613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:50737 (local address 127.19.68.62:39225)
0810 19:26:48.967787 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:48.967791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:48.968199 (+ 408us) server_negotiation.cc:407] Connection header received
0810 19:26:48.968310 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:48.968314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:48.968426 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:48.968511 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:48.969234 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.969969 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.971170 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:48.971589 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:48.971682 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:48.975615 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:50737: BlockingRecv error: recv got EOF from 127.19.68.2:50737 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:49.423204 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:49.414798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:51255)
0810 19:26:49.415338 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:49.415358 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:49.415510 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:49.415849 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:49.415853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:49.416235 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:49.416424 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.416433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.417532 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.417537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.418575 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.418588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.419467 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.419473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.419639 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.420211 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:49.420229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:49.422976 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":287,"threads_started":1}
W20260810 19:26:49.423403 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:49.414887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:51255 (local address 127.19.68.62:39225)
0810 19:26:49.415130 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:49.415137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:49.415447 (+ 310us) server_negotiation.cc:407] Connection header received
0810 19:26:49.415632 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:49.415636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:49.415698 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:49.415800 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:49.416572 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.417398 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.418730 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.419322 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.419432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.423171 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:51255: BlockingRecv error: recv got EOF from 127.19.68.3:51255 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:49.521816 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:49.513357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:53559)
0810 19:26:49.513858 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:49.513876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:49.514026 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:49.514449 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:49.514454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:49.514990 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:49.515205 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.515216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.516470 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.516474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.517439 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.517452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.518174 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.518178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.518301 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.518846 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:49.518863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:49.521569 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":242,"threads_started":1}
W20260810 19:26:49.522092 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:49.513405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:53559 (local address 127.19.68.62:39225)
0810 19:26:49.513644 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:49.513650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:49.513972 (+ 322us) server_negotiation.cc:407] Connection header received
0810 19:26:49.514163 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:49.514168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:49.514288 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:49.514382 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:49.515372 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.516340 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.517569 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.518037 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.518148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.521818 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:53559: BlockingRecv error: recv got EOF from 127.19.68.1:53559 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:49 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:49.984546 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:49.976653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:41461)
0810 19:26:49.977223 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:49.977239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:49.977358 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:49.977665 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:49.977669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:49.978106 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:49.978296 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.978305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.979319 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.979323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.980229 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:49.980242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.980857 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.980861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:49.980992 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.981602 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:49.981619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:49.984309 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":261,"threads_started":1}
W20260810 19:26:49.984750 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:49.976823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:41461 (local address 127.19.68.62:39225)
0810 19:26:49.977003 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:49.977007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:49.977314 (+ 307us) server_negotiation.cc:407] Connection header received
0810 19:26:49.977442 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:49.977445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:49.977507 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:49.977583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:49.978419 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.979162 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.980338 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:49.980730 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:49.980817 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:49.984510 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:41461: BlockingRecv error: recv got EOF from 127.19.68.2:41461 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:50.432646 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:50.424211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:49403)
0810 19:26:50.424909 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:50.424935 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:50.425076 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:50.425392 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:50.425395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:50.425831 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:50.426021 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.426031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.426965 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.426968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.427891 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.427903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.428535 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.428539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.428665 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.429245 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:50.429263 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:50.432350 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":390,"threads_started":1}
W20260810 19:26:50.432775 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:50.424269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:49403 (local address 127.19.68.62:39225)
0810 19:26:50.424511 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:50.424518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:50.425020 (+ 502us) server_negotiation.cc:407] Connection header received
0810 19:26:50.425213 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:50.425216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:50.425279 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:50.425357 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:50.426133 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.426841 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.428002 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.428420 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.428506 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.432552 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:49403: BlockingRecv error: recv got EOF from 127.19.68.3:49403 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:50.530912 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:50.522744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:51423)
0810 19:26:50.523233 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:50.523250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:50.523392 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:50.523718 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:50.523723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:50.524117 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:50.524285 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.524294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.525395 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.525399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.526298 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.526319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.527025 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.527032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.527196 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.527773 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:50.527792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:50.530603 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
W20260810 19:26:50.531128 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:50.522798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51423 (local address 127.19.68.62:39225)
0810 19:26:50.523003 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:50.523009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:50.523342 (+ 333us) server_negotiation.cc:407] Connection header received
0810 19:26:50.523511 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:50.523515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:50.523577 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:50.523658 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:50.524446 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.525253 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.526454 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.526857 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.526966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.530863 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:51423: BlockingRecv error: recv got EOF from 127.19.68.1:51423 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:50 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:50.993513 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:50.985549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:46631)
0810 19:26:50.986058 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:50.986074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:50.986189 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:50.986454 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:50.986457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:50.986827 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:50.987007 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.987016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.987998 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.988002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.988916 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:50.988928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.989753 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.989758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:50.989890 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.990475 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:50.990493 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:50.993246 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
W20260810 19:26:50.993716 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:50.985629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:46631 (local address 127.19.68.62:39225)
0810 19:26:50.985837 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:50.985843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:50.986141 (+ 298us) server_negotiation.cc:407] Connection header received
0810 19:26:50.986277 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:50.986280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:50.986343 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:50.986421 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:50.987135 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.987872 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.989154 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:50.989611 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:50.989737 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:50.993485 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:46631: BlockingRecv error: recv got EOF from 127.19.68.2:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:26:51 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:51 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:51.446497 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:51.433685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:59809)
0810 19:26:51.434208 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:51.434230 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:51.434385 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:51.434786 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:51.434791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:51.435210 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:51.435481 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.435492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.436589 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.436595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.438072 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.438088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.438920 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.438925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.439053 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:51.443234 (+ 4181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:51.443264 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:51.446192 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
W20260810 19:26:51.446759 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:51.433860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:59809 (local address 127.19.68.62:39225)
0810 19:26:51.434073 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:51.434078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:51.434319 (+ 241us) server_negotiation.cc:407] Connection header received
0810 19:26:51.434527 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:51.434532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:51.434615 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:51.434724 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:51.435654 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.436446 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.438261 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.438759 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.438869 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:51.446502 (+ 7633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:59809: BlockingRecv error: recv got EOF from 127.19.68.3:59809 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 19:26:51 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:51 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:51.539871 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:51.531956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:59677)
0810 19:26:51.532524 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:51.532541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:51.532657 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:51.532920 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:51.532923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:51.533330 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:51.533498 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.533506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.534459 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.534463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.535533 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.535547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.536274 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.536279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.536394 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:51.536949 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:51.536966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:51.539602 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":251,"threads_started":1}
W20260810 19:26:51.539963 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:51.531970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:59677 (local address 127.19.68.62:39225)
0810 19:26:51.532189 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:51.532195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:51.532618 (+ 423us) server_negotiation.cc:407] Connection header received
0810 19:26:51.532737 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:51.532740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:51.532801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:51.532874 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:51.533611 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.534313 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.535652 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.536135 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.536228 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:51.539778 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:59677: BlockingRecv error: recv got EOF from 127.19.68.1:59677 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:26:51 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:52 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:52.002708 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:51.994514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:43329)
0810 19:26:51.995117 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:51.995133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:51.995246 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:51.995536 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:51.995540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:51.995950 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:51.996120 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.996129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.997243 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.997247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.998183 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:51.998196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.998876 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.998881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:51.998989 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:51.999520 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:51.999536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:52.002423 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":272,"threads_started":1}
W20260810 19:26:52.002905 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:51.994602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:43329 (local address 127.19.68.62:39225)
0810 19:26:51.994766 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:51.994771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:51.995212 (+ 441us) server_negotiation.cc:407] Connection header received
0810 19:26:51.995337 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:51.995341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:51.995421 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:51.995505 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:51.996304 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.997102 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.998331 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:51.998756 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:51.998859 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:52.002647 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:43329: BlockingRecv error: recv got EOF from 127.19.68.2:43329 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 19:26:52 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:52 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:52.456358 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:52.447570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:51965)
0810 19:26:52.448184 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:52.448205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:52.448329 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:52.448601 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:52.448605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:52.449014 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:52.449239 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:52.449250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.450242 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.450248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:52.451575 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:52.451588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.452318 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.452322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:52.452437 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:52.453001 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:52.453019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:52.456036 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":285,"threads_started":1}
W20260810 19:26:52.456468 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:52.447651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:51965 (local address 127.19.68.62:39225)
0810 19:26:52.447812 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:52.447817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:52.448272 (+ 455us) server_negotiation.cc:407] Connection header received
0810 19:26:52.448423 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:52.448427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:52.448488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:52.448568 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:52.449412 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.450123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.451702 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.452162 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.452254 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:52.456242 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:51965: BlockingRecv error: recv got EOF from 127.19.68.3:51965 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 19:26:52 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:52 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:52.548851 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:52.540783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:43719)
0810 19:26:52.541413 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:52.541430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:52.541547 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:52.541828 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:52.541831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:52.542218 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:52.542378 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:52.542387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.543351 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.543355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:52.544259 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:52.544270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.544953 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.544957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:52.545066 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:52.545775 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:52.545795 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:52.548605 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":342,"threads_started":1}
W20260810 19:26:52.549027 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:52.540974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:43719 (local address 127.19.68.62:39225)
0810 19:26:52.541257 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:52.541263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:52.541510 (+ 247us) server_negotiation.cc:407] Connection header received
0810 19:26:52.541635 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:52.541638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:52.541700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:52.541776 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:52.542486 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.543210 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.544377 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:52.544806 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:52.544894 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:52.548797 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:43719: BlockingRecv error: recv got EOF from 127.19.68.1:43719 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:53.012426 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:53.003871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:46179)
0810 19:26:53.004403 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:53.004421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:53.004542 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:53.004879 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:53.004884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:53.005348 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:53.005565 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.005577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.006757 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.006761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.007808 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.007822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.008567 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.008572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.008689 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.009364 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:53.009385 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:53.012189 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
W20260810 19:26:53.012591 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:53.003976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:46179 (local address 127.19.68.62:39225)
0810 19:26:53.004192 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:53.004199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:53.004500 (+ 301us) server_negotiation.cc:407] Connection header received
0810 19:26:53.004633 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:53.004636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:53.004699 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:53.004775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:53.005703 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.006600 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.007923 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.008365 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.008473 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.012366 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:46179: BlockingRecv error: recv got EOF from 127.19.68.2:46179 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:53.465778 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:53.457346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:42559)
0810 19:26:53.457908 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:53.457929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:53.458050 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:53.458354 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:53.458357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:53.458739 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:53.458919 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.458928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.460018 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.460023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.461079 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.461130 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.462035 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.462039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.462154 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.462748 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:53.462767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:53.465472 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":289,"threads_started":1}
W20260810 19:26:53.465935 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:53.457492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:42559 (local address 127.19.68.62:39225)
0810 19:26:53.457671 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:53.457676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:53.458014 (+ 338us) server_negotiation.cc:407] Connection header received
0810 19:26:53.458145 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:53.458149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:53.458217 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:53.458323 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:53.459050 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.459829 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.461285 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.461719 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.461791 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.465697 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:42559: BlockingRecv error: recv got EOF from 127.19.68.3:42559 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:53 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:53.558411 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:53.549825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:53323)
0810 19:26:53.550374 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:53.550390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:53.550522 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:53.550854 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:53.550859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:53.551281 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:53.551449 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.551459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.552519 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.552523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.553504 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:53.553517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.554246 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.554252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:53.554425 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.554986 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:53.555004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:53.558156 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
W20260810 19:26:53.558617 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:53.549950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:53323 (local address 127.19.68.62:39225)
0810 19:26:53.550171 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:53.550178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:53.550476 (+ 298us) server_negotiation.cc:407] Connection header received
0810 19:26:53.550637 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:53.550641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:53.550701 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:53.550784 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:53.551608 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.552359 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.553653 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:53.554089 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:53.554199 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:53.558394 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:53323: BlockingRecv error: recv got EOF from 127.19.68.1:53323 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:54.022257 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:54.013383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:43387)
0810 19:26:54.013900 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:54.013919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:54.014038 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:54.014301 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:54.014303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:54.014726 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:54.014901 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.014910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.015952 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.015956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.016875 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.016885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.017756 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.017762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.017999 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.018764 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:54.018793 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:54.021939 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260810 19:26:54.022441 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:54.013543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:43387 (local address 127.19.68.62:39225)
0810 19:26:54.013804 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:54.013810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:54.013994 (+ 184us) server_negotiation.cc:407] Connection header received
0810 19:26:54.014131 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:54.014134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:54.014199 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:54.014287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:54.015047 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.015843 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.016990 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.017589 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.017713 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.022184 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:43387: BlockingRecv error: recv got EOF from 127.19.68.2:43387 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:54.475369 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:54.466776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:52691)
0810 19:26:54.467325 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:54.467346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:54.467475 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:54.467791 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:54.467794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:54.468188 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:54.468351 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.468361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.469443 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.469446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.470392 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.470412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.471217 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.471223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.471431 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.472015 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:54.472033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:54.475082 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":295,"threads_started":1}
W20260810 19:26:54.475481 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:54.466913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:52691 (local address 127.19.68.62:39225)
0810 19:26:54.467091 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:54.467096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:54.467459 (+ 363us) server_negotiation.cc:407] Connection header received
0810 19:26:54.467591 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:54.467595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:54.467665 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:54.467768 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:54.468523 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.469327 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.470550 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.471055 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.471167 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.475279 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:52691: BlockingRecv error: recv got EOF from 127.19.68.3:52691 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:54 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:54.568202 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:54.559474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:39197)
0810 19:26:54.560036 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:54.560058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:54.560182 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:54.560469 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:54.560472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:54.560905 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:54.561114 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.561129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.562166 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.562170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.563188 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:54.563204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.563916 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.563920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:54.564051 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.564661 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:54.564680 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:54.567887 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":311,"threads_started":1}
W20260810 19:26:54.568336 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:54.559551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:39197 (local address 127.19.68.62:39225)
0810 19:26:54.559736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:54.559740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:54.560141 (+ 401us) server_negotiation.cc:407] Connection header received
0810 19:26:54.560281 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:54.560284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:54.560347 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:54.560425 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:54.561271 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.562020 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.563318 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:54.563765 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:54.563853 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:54.568083 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:39197: BlockingRecv error: recv got EOF from 127.19.68.1:39197 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:55.031914 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:55.023217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:44329)
0810 19:26:55.023730 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:55.023746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:55.023875 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:55.024225 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:55.024229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:55.024616 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:55.024822 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.024835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.025993 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.025998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.026928 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.026941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.027669 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.027675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.027827 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.028574 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:55.028595 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:55.031684 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
W20260810 19:26:55.032119 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:55.023361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:44329 (local address 127.19.68.62:39225)
0810 19:26:55.023567 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:55.023574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:55.023831 (+ 257us) server_negotiation.cc:407] Connection header received
0810 19:26:55.024018 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:55.024024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:55.024088 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:55.024169 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:55.024966 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.025858 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.027071 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.027507 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.027633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.031881 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:44329: BlockingRecv error: recv got EOF from 127.19.68.2:44329 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:55.484685 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:55.476469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:48189)
0810 19:26:55.476970 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:55.476995 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:55.477148 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:55.477479 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:55.477484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:55.477902 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:55.478063 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.478073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.479079 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.479085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.480021 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.480033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.480776 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.480781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.480952 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.481570 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:55.481588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:55.484352 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
W20260810 19:26:55.484827 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:55.476614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:48189 (local address 127.19.68.62:39225)
0810 19:26:55.476833 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:55.476839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:55.477112 (+ 273us) server_negotiation.cc:407] Connection header received
0810 19:26:55.477267 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:55.477273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:55.477344 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:55.477423 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:55.478214 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.478942 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.480161 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.480610 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.480725 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.484602 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:48189: BlockingRecv error: recv got EOF from 127.19.68.3:48189 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:55 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:55.577911 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:55.569221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:40791)
0810 19:26:55.569703 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:55.569720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:55.569842 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:55.570201 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:55.570205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:55.570647 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:55.570835 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.570846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.571933 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.571939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.572924 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:55.572941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.573951 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.573955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:55.574087 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.574656 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:55.574674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:55.577563 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
W20260810 19:26:55.578048 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:55.569286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40791 (local address 127.19.68.62:39225)
0810 19:26:55.569549 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:55.569555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:55.569827 (+ 272us) server_negotiation.cc:407] Connection header received
0810 19:26:55.569964 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:55.569968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:55.570036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:55.570113 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:55.570991 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.571752 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.573079 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:55.573782 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:55.573885 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:55.577809 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:40791: BlockingRecv error: recv got EOF from 127.19.68.1:40791 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:56.040971 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:56.032733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:34561)
0810 19:26:56.033288 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:56.033304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:56.033430 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:56.033694 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:56.033697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:56.034106 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:56.034281 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.034293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.035360 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.035364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.036330 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.036344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.037114 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.037122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.037253 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.037836 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:56.037861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:56.040735 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":325,"threads_started":1}
W20260810 19:26:56.041181 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:56.032872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:34561 (local address 127.19.68.62:39225)
0810 19:26:56.033036 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:56.033041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:56.033372 (+ 331us) server_negotiation.cc:407] Connection header received
0810 19:26:56.033522 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:56.033525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:56.033589 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:56.033668 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:56.034406 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.035199 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.036485 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.036922 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.037022 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.040926 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:34561: BlockingRecv error: recv got EOF from 127.19.68.2:34561 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:56.502588 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:56.485688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:45507)
0810 19:26:56.486312 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:56.486337 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:56.486465 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:56.486771 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:56.486775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:56.487221 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:56.487425 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.487435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.488461 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.488466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.489535 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.489551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.490244 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.490248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.490394 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.499367 (+ 8973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:56.499390 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:56.502336 (+ 2946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":328,"threads_started":1}
W20260810 19:26:56.502810 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:56.485762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:45507 (local address 127.19.68.62:39225)
0810 19:26:56.485933 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:56.485938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:56.486412 (+ 474us) server_negotiation.cc:407] Connection header received
0810 19:26:56.486556 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:56.486559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:56.486620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:56.486697 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:56.487558 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.488321 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.489670 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.490106 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.490198 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.502572 (+ 12374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:45507: BlockingRecv error: recv got EOF from 127.19.68.3:45507 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:56 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:56.587177 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:56.578878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:45489)
0810 19:26:56.579430 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:56.579446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:56.579593 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:56.579948 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:56.579953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:56.580405 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:56.580570 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.580579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.581657 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.581662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.582589 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:56.582602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.583363 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.583368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:56.583541 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.584116 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:56.584134 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:56.586846 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":284,"threads_started":1}
W20260810 19:26:56.587306 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:56.578980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:45489 (local address 127.19.68.62:39225)
0810 19:26:56.579207 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:56.579213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:56.579534 (+ 321us) server_negotiation.cc:407] Connection header received
0810 19:26:56.579708 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:56.579713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:56.579801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:56.579884 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:56.580727 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.581507 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.582734 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:56.583214 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:56.583328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:56.587072 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:45489: BlockingRecv error: recv got EOF from 127.19.68.1:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:57.050311 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:57.041825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:41493)
0810 19:26:57.042403 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:57.042420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:57.042534 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:57.042901 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:57.042905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:57.043289 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:57.043473 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.043481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.044590 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.044594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.045559 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.045573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.046304 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.046308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.046443 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.046993 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:57.047011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:57.050018 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":317,"threads_started":1}
W20260810 19:26:57.050508 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:57.041934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:41493 (local address 127.19.68.62:39225)
0810 19:26:57.042110 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:57.042114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:57.042501 (+ 387us) server_negotiation.cc:407] Connection header received
0810 19:26:57.042676 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:57.042680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:57.042760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:57.042845 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:57.043641 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.044444 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.045693 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.046182 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.046302 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.050272 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:41493: BlockingRecv error: recv got EOF from 127.19.68.2:41493 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:57.511545 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:57.503450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:40325)
0810 19:26:57.503971 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:57.503992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:57.504125 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:57.504476 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:57.504480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:57.504860 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:57.505025 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.505033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.506122 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.506126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.507078 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.507091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.507799 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.507805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.507976 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.508553 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:57.508573 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:57.511307 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
W20260810 19:26:57.511735 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:57.503496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:40325 (local address 127.19.68.62:39225)
0810 19:26:57.503656 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:57.503662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:57.504073 (+ 411us) server_negotiation.cc:407] Connection header received
0810 19:26:57.504235 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:57.504240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:57.504317 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:57.504400 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:57.505181 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.505973 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.507214 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.507658 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.507776 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.511511 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:40325: BlockingRecv error: recv got EOF from 127.19.68.3:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:57 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:57.596346 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:57.588090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:33625)
0810 19:26:57.588598 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:57.588616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:57.588731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:57.589028 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:57.589031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:57.589515 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:57.589679 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.589690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.590621 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.590624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.591648 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:57.591660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.592347 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.592352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:57.592472 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.592983 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:57.593000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:57.596107 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":241,"threads_started":1}
W20260810 19:26:57.596510 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:57.588194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:33625 (local address 127.19.68.62:39225)
0810 19:26:57.588421 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:57.588428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:57.588685 (+ 257us) server_negotiation.cc:407] Connection header received
0810 19:26:57.588854 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:57.588857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:57.588923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:57.589001 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:57.589800 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.590517 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.591767 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:57.592189 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:57.592287 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:57.596297 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:33625: BlockingRecv error: recv got EOF from 127.19.68.1:33625 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:58.059481 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:58.051159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:41995)
0810 19:26:58.051816 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:58.051833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:58.051965 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:58.052319 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:58.052323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:58.052914 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:58.053112 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.053123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.054193 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.054196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.055106 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.055117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.055824 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.055828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.055955 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.056534 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:58.056553 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:58.059224 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":395,"threads_started":1}
W20260810 19:26:58.059646 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:58.051386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:41995 (local address 127.19.68.62:39225)
0810 19:26:58.051570 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:58.051576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:58.051923 (+ 347us) server_negotiation.cc:407] Connection header received
0810 19:26:58.052085 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:58.052090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:58.052160 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:58.052276 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:58.053298 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.054074 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.055249 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.055680 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.055777 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.059442 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:41995: BlockingRecv error: recv got EOF from 127.19.68.2:41995 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:58.520814 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:58.512485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:48623)
0810 19:26:58.513006 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:58.513026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:58.513179 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:58.513494 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:58.513499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:58.513888 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:58.514061 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.514071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.515059 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.515063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.515989 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.516001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.516720 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.516726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.516874 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.517533 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:58.517558 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:58.520561 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":239,"threads_started":1}
W20260810 19:26:58.520962 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:58.512550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:48623 (local address 127.19.68.62:39225)
0810 19:26:58.512729 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:58.512735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:58.513120 (+ 385us) server_negotiation.cc:407] Connection header received
0810 19:26:58.513290 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:58.513295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:58.513358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:58.513443 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:58.514203 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.514922 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.516122 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.516544 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.516650 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.520755 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:48623: BlockingRecv error: recv got EOF from 127.19.68.3:48623 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:58 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:58.605408 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:58.597246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:51015)
0810 19:26:58.597725 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:58.597742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:58.597854 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:58.598138 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:58.598141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:58.598545 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:58.598721 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.598731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.599717 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.599721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.600616 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:58.600628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.601304 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.601308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:58.601449 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.601988 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:58.602004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:58.605026 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
W20260810 19:26:58.605511 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:58.597367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51015 (local address 127.19.68.62:39225)
0810 19:26:58.597615 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:58.597620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:58.597804 (+ 184us) server_negotiation.cc:407] Connection header received
0810 19:26:58.597937 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:58.597940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:58.598018 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:58.598104 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:58.598845 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.599596 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.600736 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:58.601193 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:58.601291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:58.605269 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:51015: BlockingRecv error: recv got EOF from 127.19.68.1:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:59.068167 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:59.060494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:33245)
0810 19:26:59.060919 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:59.060936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:59.061048 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:59.061334 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:59.061337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:59.061699 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:59.061875 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.061884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.062861 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.062865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.063781 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.063793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.064521 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.064524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.064640 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.065267 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:59.065284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:59.067919 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
W20260810 19:26:59.068379 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:59.060540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:33245 (local address 127.19.68.62:39225)
0810 19:26:59.060743 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:59.060749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:59.061007 (+ 258us) server_negotiation.cc:407] Connection header received
0810 19:26:59.061166 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:59.061169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:59.061239 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:59.061321 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:59.061999 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.062742 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.063929 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.064395 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.064502 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.068127 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:33245: BlockingRecv error: recv got EOF from 127.19.68.2:33245 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.3@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:59.529657 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:59.521696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.3:43937)
0810 19:26:59.522162 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:59.522184 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:59.522319 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:59.522637 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:59.522641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:59.523029 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:59.523202 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.523212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.524239 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.524243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.525179 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.525191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.525847 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.525853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.526031 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.526605 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:59.526623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:59.529290 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
W20260810 19:26:59.529791 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:59.521786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:43937 (local address 127.19.68.62:39225)
0810 19:26:59.522006 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:59.522012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:59.522275 (+ 263us) server_negotiation.cc:407] Connection header received
0810 19:26:59.522433 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:59.522438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:59.522507 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:59.522590 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:59.523353 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.524098 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.525316 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.525708 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.525825 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.529556 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.3:43937: BlockingRecv error: recv got EOF from 127.19.68.3:43937 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:26:59 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.1@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:26:59.614581 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:26:59.606586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.1:52363)
0810 19:26:59.607123 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 19:26:59.607140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:26:59.607275 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:26:59.607630 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:26:59.607633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:26:59.608063 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:26:59.608308 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.608321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.609320 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.609324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.610240 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:26:59.610252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.610885 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.610888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:26:59.611001 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.611583 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:26:59.611601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:26:59.614328 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":341,"threads_started":1}
W20260810 19:26:59.614732 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:26:59.606654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:52363 (local address 127.19.68.62:39225)
0810 19:26:59.606946 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 19:26:59.606953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:26:59.607233 (+ 280us) server_negotiation.cc:407] Connection header received
0810 19:26:59.607414 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:26:59.607419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:26:59.607503 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:26:59.607581 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:26:59.608449 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.609217 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.610370 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:26:59.610778 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:26:59.610864 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:26:59.614519 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.1:52363: BlockingRecv error: recv got EOF from 127.19.68.1:52363 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 10 19:27:00 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:27:00 dist-test-slave-2j7r krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.68.2@KRBTEST.COM for oryx/127.19.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:27:00.077661 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:00.069239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39225 (local address 127.19.68.2:58635)
0810 19:27:00.069751 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:00.069770 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:00.069904 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:00.070247 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:00.070291 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:00.070703 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:00.070904 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:00.070914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.071954 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.071958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:00.072890 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:00.072902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.073699 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.073705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:00.073906 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:00.074443 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:00.074462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:00.077304 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.68.62:39225: Server oryx/127.19.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
W20260810 19:27:00.077788 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:27:00.069315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:58635 (local address 127.19.68.62:39225)
0810 19:27:00.069523 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:00.069529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:00.069857 (+ 328us) server_negotiation.cc:407] Connection header received
0810 19:27:00.070019 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:00.070024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:00.070092 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:00.070175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:00.071055 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.071796 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.073032 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.073551 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.073669 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:00.077535 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.68.2:58635: BlockingRecv error: recv got EOF from 127.19.68.2:58635 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:27:00.359994 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2675
I20260810 19:27:00.371603 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:00.361022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.19.68.1:37131)
0810 19:27:00.361593 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:00.361600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:00.361626 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:27:00.361682 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:00.361688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:00.361929 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:00.362077 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:00.363476 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.364460 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.365682 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.366255 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.366350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:00.368694 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:00.368703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:00.368715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:00.368746 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:00.371139 (+ 2393us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:27:00.371380 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59316: Request ticket server kudu/127.19.68.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
W20260810 19:27:00.371698 3590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59316: Request ticket server kudu/127.19.68.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:27:00.371770 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2675: Not authorized: Client connection negotiation failed: client connection to 127.19.68.1:37131: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:27:00.372009 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 2675
I20260810 19:27:00.396997 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2977
I20260810 19:27:00.409335 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:00.398188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.19.68.2:42079)
0810 19:27:00.398792 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:00.398800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:00.398830 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:27:00.398891 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:00.398915 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:00.399164 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:00.399311 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:00.400560 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.401850 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.403122 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.403745 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.403859 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:00.406442 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:00.406450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:00.406463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:00.406498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:00.408809 (+ 2311us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:27:00.409056 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58600: Request ticket server kudu/127.19.68.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":282,"threads_started":1}
W20260810 19:27:00.409442 3592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58600: Request ticket server kudu/127.19.68.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:27:00.409533 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2977: Not authorized: Client connection negotiation failed: client connection to 127.19.68.2:42079: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:27:00.409686 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 2977
I20260810 19:27:00.444320 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 3251
I20260810 19:27:00.456014 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:00.445409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.19.68.3:43755)
0810 19:27:00.445912 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:00.445919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:00.445947 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:27:00.446032 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:00.446039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:00.446259 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:00.446429 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:00.447695 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.448652 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.449934 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:00.450468 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:00.450561 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:00.452933 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:00.452942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:00.452955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:00.452990 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:00.455517 (+ 2527us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:27:00.455751 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50748: Request ticket server kudu/127.19.68.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
W20260810 19:27:00.456108 3594 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50748: Request ticket server kudu/127.19.68.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:27:00.456162 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 3251: Not authorized: Client connection negotiation failed: client connection to 127.19.68.3:43755: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.68.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:27:00.456326 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 3251
I20260810 19:27:00.480244 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 2550
I20260810 19:27:00.481585 2666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:00.597834 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 2550
2026-08-10T19:27:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21365 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T19:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:27:00Z Disabled control of system clock
I20260810 19:27:00.694563 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39943
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 19:27:01.028863 3603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:01.029158 3603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:01.029248 3603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:01.036293 3603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:27:01.036386 3603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:01.036422 3603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:01.036476 3603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:27:01.036520 3603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:27:01.045398 3603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:01.046734 3603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:01.048501 3603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:01.060084 3615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:01.060204 3613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:01.060310 3612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:01.060727 3603 server_base.cc:1034] running on GCE node
I20260810 19:27:01.061558 3603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:01.062942 3603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:01.064174 3603 hybrid_clock.cc:648] HybridClock initialized: now 1786390021064125 us; error 42 us; skew 500 ppm
I20260810 19:27:01.064536 3603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:01.066757 3603 webserver.cc:460] Webserver started at http://127.19.68.62:46499/ using document root <none> and password file <none>
I20260810 19:27:01.067430 3603 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:01.067529 3603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:01.067842 3603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:01.070705 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "b94e3f175e974d258735e14a6101421c"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "22c6ac40b99e6df6d21d84be5f976def"
server_key_iv: "4c10ee2869d0ee57e95340384b104b05"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.071455 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "b94e3f175e974d258735e14a6101421c"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "22c6ac40b99e6df6d21d84be5f976def"
server_key_iv: "4c10ee2869d0ee57e95340384b104b05"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.076442 3603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:27:01.079772 3626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:01.081017 3603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:27:01.081197 3603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "b94e3f175e974d258735e14a6101421c"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "22c6ac40b99e6df6d21d84be5f976def"
server_key_iv: "4c10ee2869d0ee57e95340384b104b05"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.081385 3603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:01.096570 3603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:01.097405 3603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:01.097656 3603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:01.114984 3603 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39943
I20260810 19:27:01.115003 3735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39943 every 8 connection(s)
I20260810 19:27:01.116290 3603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:27:01.117431 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 3603
I20260810 19:27:01.117625 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:27:01.118450 19728 external_mini_cluster.cc:1428] Setting key 08ec866a93b447dcf837ae9475bd47c5
I20260810 19:27:01.120973 3738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:01.132174 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:01.120602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.19.68.62:39943)
0810 19:27:01.121572 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:01.121582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:01.121622 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:27:01.123195 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:01.123217 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:01.123655 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:01.123986 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:01.124707 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.126114 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.127924 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.128556 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.128681 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:01.129939 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:01.129947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:01.129960 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:01.130493 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:01.130582 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:01.130606 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:01.130696 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:01.130919 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 19:27:01.131221 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":393,"threads_started":1}
I20260810 19:27:01.133558 3738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c: Bootstrap starting.
I20260810 19:27:01.136610 3738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:01.137602 3738 log.cc:824] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:01.140429 3738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c: No bootstrap required, opened a new log
I20260810 19:27:01.147264 3738 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } }
I20260810 19:27:01.147595 3738 raft_consensus.cc:374] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:01.147678 3738 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94e3f175e974d258735e14a6101421c, State: Initialized, Role: FOLLOWER
I20260810 19:27:01.148205 3738 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } }
I20260810 19:27:01.148430 3738 raft_consensus.cc:388] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:27:01.148552 3738 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:27:01.148708 3738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:01.150748 3738 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } }
I20260810 19:27:01.151188 3738 leader_election.cc:302] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b94e3f175e974d258735e14a6101421c; no voters:
I20260810 19:27:01.151741 3738 leader_election.cc:288] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:27:01.151899 3753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:01.152792 3753 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [term 1 LEADER]: Becoming Leader. State: Replica: b94e3f175e974d258735e14a6101421c, State: Running, Role: LEADER
I20260810 19:27:01.153334 3753 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } }
I20260810 19:27:01.153806 3738 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:27:01.156092 3754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b94e3f175e974d258735e14a6101421c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } } }
I20260810 19:27:01.156296 3755 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94e3f175e974d258735e14a6101421c. Latest consensus state: current_term: 1 leader_uuid: "b94e3f175e974d258735e14a6101421c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e3f175e974d258735e14a6101421c" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39943 } } }
I20260810 19:27:01.156580 3754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:01.156667 3755 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:01.162338 3770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:27:01.166409 3770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:27:01.173925 3770 catalog_manager.cc:1347] Generated new cluster ID: 55391b59ee9a4eeab938b3c5d26f0036
I20260810 19:27:01.174034 3770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:27:01.194860 3770 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:27:01.196069 3770 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:27:01.201561 3770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94e3f175e974d258735e14a6101421c: Generated new TSK 0
I20260810 19:27:01.202349 3770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:27:01.208904 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--builtin_ntp_servers=127.19.68.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 19:27:01.536953 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:01.537233 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:01.537334 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:01.544342 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:01.544441 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:01.544541 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:27:01.553339 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:01.554652 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:01.556421 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:01.569069 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:01.569123 3799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:01.569123 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:01.569669 3789 server_base.cc:1034] running on GCE node
I20260810 19:27:01.570336 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:01.571241 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:01.572443 3789 hybrid_clock.cc:648] HybridClock initialized: now 1786390021572398 us; error 35 us; skew 500 ppm
I20260810 19:27:01.572734 3789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:01.579337 3789 webserver.cc:460] Webserver started at http://127.19.68.1:45123/ using document root <none> and password file <none>
I20260810 19:27:01.580031 3789 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:01.580139 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:01.580470 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:01.583302 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "91d70da752234b87a5416dff563cadc2"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "825d15604190869c439517140a1e1d72"
server_key_iv: "bfd0efc0717ddbae21a34f6053ec8bb1"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.584084 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "91d70da752234b87a5416dff563cadc2"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "825d15604190869c439517140a1e1d72"
server_key_iv: "bfd0efc0717ddbae21a34f6053ec8bb1"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.589126 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:27:01.592408 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:01.593698 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:27:01.593851 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "91d70da752234b87a5416dff563cadc2"
format_stamp: "Formatted at 2026-08-10 19:27:01 on dist-test-slave-2j7r"
server_key: "825d15604190869c439517140a1e1d72"
server_key_iv: "bfd0efc0717ddbae21a34f6053ec8bb1"
server_key_version: "encryptionkey@0"
I20260810 19:27:01.594044 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:01.610724 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:01.611482 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:01.611728 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:01.612607 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:01.613986 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:01.614081 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:01.614188 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:01.614260 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:01.644788 3789 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:34875
I20260810 19:27:01.644824 4048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:34875 every 8 connection(s)
I20260810 19:27:01.646149 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:27:01.647795 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 3789
I20260810 19:27:01.648007 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:27:01.648732 19728 external_mini_cluster.cc:1428] Setting key a8773f4a6bbaacb669bf3d3e20343758
I20260810 19:27:01.651441 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--builtin_ntp_servers=127.19.68.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 19:27:01.659819 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:01.648957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40243 (local address 127.19.68.62:39943)
0810 19:27:01.649489 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:01.649495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:01.650813 (+ 1318us) server_negotiation.cc:407] Connection header received
0810 19:27:01.652554 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:01.652560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:01.652637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:01.652746 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:01.653808 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.654738 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.656784 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.657417 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.657563 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:01.658931 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:01.658938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:01.658944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:01.658968 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:01.659022 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:01.659041 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:01.659140 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:01.659335 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 19:27:01.659440 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260810 19:27:01.660714 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:01.649546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39943 (local address 127.19.68.1:40243)
0810 19:27:01.650634 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:01.650687 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:01.652302 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:01.652930 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:01.652938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:01.653019 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:01.653613 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:01.653630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.654967 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.654972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:01.656622 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:01.656640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:01.657632 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:01.657638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:01.657893 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:01.658583 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:01.658586 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:01.658630 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:01.658710 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:01.659130 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:01.659137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:01.659143 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:27:01.659304 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 19:27:01.659636 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":346,"threads_started":1}
I20260810 19:27:01.662442 4051 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39943
I20260810 19:27:01.662777 4051 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:01.663759 4051 heartbeater.cc:511] Master 127.19.68.62:39943 requested a full tablet report, sending...
I20260810 19:27:01.665905 3675 ts_manager.cc:194] Registered new tserver with Master: 91d70da752234b87a5416dff563cadc2 (127.19.68.1:34875)
I20260810 19:27:01.667533 3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.1:40243
W20260810 19:27:01.980357 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:01.980629 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:01.980754 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:01.987998 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:01.988092 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:01.988240 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:27:01.997483 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:01.998787 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:02.000557 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:02.013304 4064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:02.013245 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:02.013250 4065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:02.014073 4058 server_base.cc:1034] running on GCE node
I20260810 19:27:02.014714 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:02.015636 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:02.016865 4058 hybrid_clock.cc:648] HybridClock initialized: now 1786390022016805 us; error 54 us; skew 500 ppm
I20260810 19:27:02.017253 4058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:02.019506 4058 webserver.cc:460] Webserver started at http://127.19.68.2:42245/ using document root <none> and password file <none>
I20260810 19:27:02.020180 4058 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:02.020291 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:02.020592 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:02.023384 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "4ee7caf83a20434abd6b68b772b1e329"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "8239a546d64ebf715886e01ce0f2cb6b"
server_key_iv: "0d39234e2f2d5f51b36605c474a4a4bf"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.024150 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "4ee7caf83a20434abd6b68b772b1e329"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "8239a546d64ebf715886e01ce0f2cb6b"
server_key_iv: "0d39234e2f2d5f51b36605c474a4a4bf"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.029114 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:27:02.032351 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.033663 4058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:27:02.033809 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "4ee7caf83a20434abd6b68b772b1e329"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "8239a546d64ebf715886e01ce0f2cb6b"
server_key_iv: "0d39234e2f2d5f51b36605c474a4a4bf"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.033998 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:02.049449 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:02.050201 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:02.050452 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:02.051362 4058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:02.052697 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:02.052796 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.052901 4058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:02.052969 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.083858 4058 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:46759
I20260810 19:27:02.083909 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:46759 every 8 connection(s)
I20260810 19:27:02.085223 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:27:02.089497 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 4058
I20260810 19:27:02.089635 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:27:02.090075 19728 external_mini_cluster.cc:1428] Setting key a8138f6cfc64955b72acca36cad8e141
I20260810 19:27:02.093222 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--builtin_ntp_servers=127.19.68.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 19:27:02.097903 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.087661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:59145 (local address 127.19.68.62:39943)
0810 19:27:02.087863 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.087867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.089241 (+ 1374us) server_negotiation.cc:407] Connection header received
0810 19:27:02.091177 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.091181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.091241 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.091321 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.092596 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.093617 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.095164 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.095696 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.095811 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.097120 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:02.097126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:02.097130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:02.097151 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:02.097186 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:02.097201 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:02.097284 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:02.097498 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 19:27:02.097593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:27:02.098954 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.088132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39943 (local address 127.19.68.2:59145)
0810 19:27:02.089042 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:02.089118 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:02.090934 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:02.091569 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:02.091579 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:02.091688 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:02.092433 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:02.092453 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.093777 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.093781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:02.095018 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:02.095031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.095897 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.095902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:02.096107 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.096759 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:02.096762 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:02.096801 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:02.096894 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:02.097304 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:02.097315 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:02.097321 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:27:02.097477 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 19:27:02.097806 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":293,"threads_started":1}
I20260810 19:27:02.100381 4300 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39943
I20260810 19:27:02.100682 4300 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:02.101634 4300 heartbeater.cc:511] Master 127.19.68.62:39943 requested a full tablet report, sending...
I20260810 19:27:02.103161 3675 ts_manager.cc:194] Registered new tserver with Master: 4ee7caf83a20434abd6b68b772b1e329 (127.19.68.2:46759)
I20260810 19:27:02.104130 3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.2:59145
W20260810 19:27:02.422077 4304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:02.422338 4304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:02.422462 4304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:02.429625 4304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:02.429726 4304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:02.429880 4304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:27:02.439167 4304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:02.440523 4304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:02.442359 4304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:02.454794 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:02.454870 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:02.454823 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:02.455649 4304 server_base.cc:1034] running on GCE node
I20260810 19:27:02.456267 4304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:02.457226 4304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:02.458446 4304 hybrid_clock.cc:648] HybridClock initialized: now 1786390022458385 us; error 48 us; skew 500 ppm
I20260810 19:27:02.458755 4304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:02.461025 4304 webserver.cc:460] Webserver started at http://127.19.68.3:39855/ using document root <none> and password file <none>
I20260810 19:27:02.461707 4304 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:02.461809 4304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:02.462131 4304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:02.464897 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "c47980e53abd476181c19f0d697d78f2"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "1b0d5df8e082bbfa8c6b2652e4ebcf68"
server_key_iv: "677ade907a2bdadbae0c15fde48cfc93"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.465706 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "c47980e53abd476181c19f0d697d78f2"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "1b0d5df8e082bbfa8c6b2652e4ebcf68"
server_key_iv: "677ade907a2bdadbae0c15fde48cfc93"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.470618 4304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:27:02.473894 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.475148 4304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:27:02.475328 4304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "c47980e53abd476181c19f0d697d78f2"
format_stamp: "Formatted at 2026-08-10 19:27:02 on dist-test-slave-2j7r"
server_key: "1b0d5df8e082bbfa8c6b2652e4ebcf68"
server_key_iv: "677ade907a2bdadbae0c15fde48cfc93"
server_key_version: "encryptionkey@0"
I20260810 19:27:02.475526 4304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:02.491900 4304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:02.492667 4304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:02.492924 4304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:02.493887 4304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:02.495226 4304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:02.495332 4304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.495436 4304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:02.495507 4304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:02.524940 4304 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:41009
I20260810 19:27:02.525039 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:41009 every 8 connection(s)
I20260810 19:27:02.526230 4304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:27:02.531878 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 4304
I20260810 19:27:02.532049 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:27:02.532574 19728 external_mini_cluster.cc:1428] Setting key 312777d2caa891d0a6410c78cec1e542
I20260810 19:27:02.538283 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.528628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:60285 (local address 127.19.68.62:39943)
0810 19:27:02.528801 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.528806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.530202 (+ 1396us) server_negotiation.cc:407] Connection header received
0810 19:27:02.531959 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.531965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.532059 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.532174 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.533228 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.534192 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.535866 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.536303 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.536405 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.537613 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:02.537618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:02.537621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:02.537638 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:02.537668 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:02.537680 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:02.537744 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:02.537953 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 19:27:02.538032 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:27:02.539249 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.529168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39943 (local address 127.19.68.3:60285)
0810 19:27:02.530047 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:02.530081 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:02.531626 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:02.532353 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:02.532360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:02.532458 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:02.533039 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:02.533056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.534338 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.534342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:02.535730 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:02.535743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.536485 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.536490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:02.536684 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.537334 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:02.537336 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:02.537373 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:02.537455 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:02.537767 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:02.537777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:02.537785 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:27:02.537920 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 19:27:02.538217 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":277,"threads_started":1}
I20260810 19:27:02.540562 4525 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39943
I20260810 19:27:02.540841 4525 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:02.541656 4525 heartbeater.cc:511] Master 127.19.68.62:39943 requested a full tablet report, sending...
I20260810 19:27:02.543064 3675 ts_manager.cc:194] Registered new tserver with Master: c47980e53abd476181c19f0d697d78f2 (127.19.68.3:41009)
I20260810 19:27:02.543880 3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.3:60285
I20260810 19:27:02.547794 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:27:02.553896 4056 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:27:02.552375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.19.68.62:39943)
0810 19:27:02.552599 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.552606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.552936 (+ 330us) server_negotiation.cc:407] Connection header received
0810 19:27:02.553065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.553068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.553164 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.553240 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.553672 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56958: BlockingRecv error: recv got EOF from 127.0.0.1:56958 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:27:02.554653 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 3789
I20260810 19:27:02.562144 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.555578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.19.68.1:34875)
0810 19:27:02.556059 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.556066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.556091 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:27:02.556149 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.556155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.556346 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.556471 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.556885 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.557913 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.559105 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.559624 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.559730 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.560638 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:02.560646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:02.560657 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:02.561344 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:02.561392 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:02.561420 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:02.561505 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:02.561647 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 19:27:02.561773 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":211,"threads_started":1}
I20260810 19:27:02.563354 4039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:02.644047 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 3789
I20260810 19:27:02.665163 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 4058
I20260810 19:27:02.672856 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.666076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.19.68.2:46759)
0810 19:27:02.666642 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.666649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.666677 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:27:02.666737 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.666743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.666975 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.667099 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.667466 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.668450 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.669793 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.670365 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.670463 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.671406 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:02.671415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:02.671428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:02.672029 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:02.672078 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:02.672098 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:02.672191 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:02.672375 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 19:27:02.672491 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
I20260810 19:27:02.674132 4291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:02.756016 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 4058
I20260810 19:27:02.784307 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 4304
I20260810 19:27:02.792539 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:02.785417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.19.68.3:41009)
0810 19:27:02.786098 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:02.786105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:02.786191 (+ 86us) server_negotiation.cc:407] Connection header received
0810 19:27:02.786358 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:02.786366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:02.786603 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:02.786730 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:02.787084 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.788105 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.789531 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:02.790092 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:02.790189 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:02.791195 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:02.791204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:02.791217 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:02.791800 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:02.791851 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:02.791871 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:02.791959 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:02.792098 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 19:27:02.792219 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":354,"threads_started":1}
I20260810 19:27:02.793846 4516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:02.878259 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 4304
I20260810 19:27:02.903018 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 3603
I20260810 19:27:02.904421 3729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:03.006218 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 3603
2026-08-10T19:27:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2399 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T19:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:27:03Z Disabled control of system clock
I20260810 19:27:03.096570 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40875
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 19:27:03.422801 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:03.423058 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:03.423146 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:03.430099 4567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:27:03.430218 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:03.430289 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:03.430363 4567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:27:03.430440 4567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:27:03.439824 4567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:40875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:03.441112 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:03.443009 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:03.455158 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:03.455120 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:03.455195 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:03.455891 4567 server_base.cc:1034] running on GCE node
I20260810 19:27:03.456632 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:03.458010 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:03.459219 4567 hybrid_clock.cc:648] HybridClock initialized: now 1786390023459168 us; error 43 us; skew 500 ppm
I20260810 19:27:03.459544 4567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:03.461630 4567 webserver.cc:460] Webserver started at http://127.19.68.62:41143/ using document root <none> and password file <none>
I20260810 19:27:03.462271 4567 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:03.462378 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:03.462703 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:03.465497 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "19338db8e83d491fbe2459d3e471e999"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "dc41fb31f27b8bb699c35631b2b74cec"
server_key_iv: "0b95c81f5da702df90d2f0fc4cce1722"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.466274 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "19338db8e83d491fbe2459d3e471e999"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "dc41fb31f27b8bb699c35631b2b74cec"
server_key_iv: "0b95c81f5da702df90d2f0fc4cce1722"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.471045 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:27:03.474243 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:03.475457 4567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:27:03.475600 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "19338db8e83d491fbe2459d3e471e999"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "dc41fb31f27b8bb699c35631b2b74cec"
server_key_iv: "0b95c81f5da702df90d2f0fc4cce1722"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.475795 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:03.481171 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:03.481899 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:03.482167 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:03.498634 4567 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:40875
I20260810 19:27:03.498665 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:40875 every 8 connection(s)
I20260810 19:27:03.499891 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:27:03.503509 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 4567
I20260810 19:27:03.503607 4694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:03.503741 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:27:03.504550 19728 external_mini_cluster.cc:1428] Setting key f66bd11bd851a19cb3e97c1b989d66c6
I20260810 19:27:03.513459 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:03.506748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.19.68.62:40875)
0810 19:27:03.507720 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:03.507731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:03.507774 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:27:03.509804 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:03.509826 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:03.510247 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:03.510607 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:03.511705 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:03.511711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:03.511724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:03.511741 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:03.511804 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:03.511821 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:03.512052 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:03.512181 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 19:27:03.512473 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":345,"threads_started":1}
I20260810 19:27:03.516387 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999: Bootstrap starting.
I20260810 19:27:03.519423 4694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:03.520380 4694 log.cc:824] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:03.522982 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999: No bootstrap required, opened a new log
I20260810 19:27:03.530191 4694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } }
I20260810 19:27:03.530539 4694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:03.530635 4694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19338db8e83d491fbe2459d3e471e999, State: Initialized, Role: FOLLOWER
I20260810 19:27:03.531219 4694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } }
I20260810 19:27:03.531417 4694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:27:03.531544 4694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:27:03.531687 4694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:03.533805 4694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } }
I20260810 19:27:03.534276 4694 leader_election.cc:302] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19338db8e83d491fbe2459d3e471e999; no voters:
I20260810 19:27:03.534930 4694 leader_election.cc:288] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:27:03.535056 4702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:03.535800 4702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [term 1 LEADER]: Becoming Leader. State: Replica: 19338db8e83d491fbe2459d3e471e999, State: Running, Role: LEADER
I20260810 19:27:03.536365 4702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } }
I20260810 19:27:03.536908 4694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:27:03.538904 4703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19338db8e83d491fbe2459d3e471e999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } } }
I20260810 19:27:03.539361 4703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:03.539235 4704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19338db8e83d491fbe2459d3e471e999. Latest consensus state: current_term: 1 leader_uuid: "19338db8e83d491fbe2459d3e471e999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19338db8e83d491fbe2459d3e471e999" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 40875 } } }
I20260810 19:27:03.539682 4704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999 [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:03.541682 4707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:27:03.546222 4707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:27:03.554401 4707 catalog_manager.cc:1347] Generated new cluster ID: 9afd3022be6541ffafc1aa8d015c941c
I20260810 19:27:03.554498 4707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:27:03.577716 4707 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:27:03.578125 4707 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:27:03.594693 4707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19338db8e83d491fbe2459d3e471e999: Generated new TSK 0
I20260810 19:27:03.595681 4707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:27:03.610419 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--builtin_ntp_servers=127.19.68.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 19:27:03.939147 4736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:03.939437 4736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:03.939569 4736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:03.946875 4736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:03.946974 4736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:03.947129 4736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:27:03.956301 4736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:03.957623 4736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:03.959353 4736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:03.971724 4743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:03.971767 4744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:03.972054 4746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:03.972496 4736 server_base.cc:1034] running on GCE node
I20260810 19:27:03.973110 4736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:03.974056 4736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:03.975274 4736 hybrid_clock.cc:648] HybridClock initialized: now 1786390023975190 us; error 39 us; skew 500 ppm
I20260810 19:27:03.975584 4736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:03.977769 4736 webserver.cc:460] Webserver started at http://127.19.68.1:46403/ using document root <none> and password file <none>
I20260810 19:27:03.978449 4736 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:03.978547 4736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:03.978853 4736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:03.983809 4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "2d3232d066c04e4ca783cf6bd736a69e"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "9be76ae8e79ec141a121fe698859cd64"
server_key_iv: "7e443cfa34e814b30ce410f511838964"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.984579 4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "2d3232d066c04e4ca783cf6bd736a69e"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "9be76ae8e79ec141a121fe698859cd64"
server_key_iv: "7e443cfa34e814b30ce410f511838964"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.989553 4736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:27:03.992751 4755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:03.994041 4736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:27:03.994201 4736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "2d3232d066c04e4ca783cf6bd736a69e"
format_stamp: "Formatted at 2026-08-10 19:27:03 on dist-test-slave-2j7r"
server_key: "9be76ae8e79ec141a121fe698859cd64"
server_key_iv: "7e443cfa34e814b30ce410f511838964"
server_key_version: "encryptionkey@0"
I20260810 19:27:03.994423 4736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:03.999835 4736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:04.000582 4736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:04.000828 4736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:04.001725 4736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:04.003067 4736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:04.003173 4736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.003288 4736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:04.003356 4736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.033600 4736 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:40873
I20260810 19:27:04.033703 4949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:40873 every 8 connection(s)
I20260810 19:27:04.034861 4736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:27:04.039031 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 4736
I20260810 19:27:04.039223 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:27:04.039835 19728 external_mini_cluster.cc:1428] Setting key b1cd40c2cdb4eb6b8b0bd443a273e74e
I20260810 19:27:04.043102 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--builtin_ntp_servers=127.19.68.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 19:27:04.043174 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.037280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:50903 (local address 127.19.68.62:40875)
0810 19:27:04.037774 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:04.037780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:04.038910 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 19:27:04.040747 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:04.040752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:04.040819 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:04.040952 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:04.042405 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:04.042411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:04.042415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:04.042436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:04.042488 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:04.042505 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:04.042619 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:04.042809 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 19:27:04.042877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":222,"threads_started":1}
I20260810 19:27:04.044047 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.037861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40875 (local address 127.19.68.1:50903)
0810 19:27:04.038762 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:04.038797 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:04.040454 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:04.041244 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:04.041255 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:04.041367 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:04.042077 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:04.042079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:04.042117 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:04.042199 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:04.042598 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:04.042606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:04.042611 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:27:04.042849 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 19:27:04.043064 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
I20260810 19:27:04.045478 4950 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40875
I20260810 19:27:04.045802 4950 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:04.046463 4950 heartbeater.cc:511] Master 127.19.68.62:40875 requested a full tablet report, sending...
I20260810 19:27:04.048545 4636 ts_manager.cc:194] Registered new tserver with Master: 2d3232d066c04e4ca783cf6bd736a69e (127.19.68.1:40873)
W20260810 19:27:04.370131 4959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:04.370441 4959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:04.370543 4959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:04.377471 4959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:04.377556 4959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:04.377671 4959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:27:04.386550 4959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:04.387696 4959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:04.389437 4959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:04.401746 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:04.401854 4971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:04.402061 4970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:04.402920 4959 server_base.cc:1034] running on GCE node
I20260810 19:27:04.403426 4959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:04.404371 4959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:04.405562 4959 hybrid_clock.cc:648] HybridClock initialized: now 1786390024405514 us; error 45 us; skew 500 ppm
I20260810 19:27:04.405858 4959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:04.407975 4959 webserver.cc:460] Webserver started at http://127.19.68.2:37019/ using document root <none> and password file <none>
I20260810 19:27:04.408612 4959 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:04.408713 4959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:04.409025 4959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:04.411747 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "c828c390460e434188a75e4e003ae5f1"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "e374287f104b2be7b5fe716cc45857ee"
server_key_iv: "975afa7578d41026eb9e51865c93fa2a"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.412508 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "c828c390460e434188a75e4e003ae5f1"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "e374287f104b2be7b5fe716cc45857ee"
server_key_iv: "975afa7578d41026eb9e51865c93fa2a"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.417303 4959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 19:27:04.420440 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.421727 4959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:27:04.421875 4959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "c828c390460e434188a75e4e003ae5f1"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "e374287f104b2be7b5fe716cc45857ee"
server_key_iv: "975afa7578d41026eb9e51865c93fa2a"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.422065 4959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:04.427366 4959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:04.428071 4959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:04.428300 4959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:04.429157 4959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:04.430430 4959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:04.430526 4959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.430627 4959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:04.430697 4959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.460494 4959 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:39061
I20260810 19:27:04.460547 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:39061 every 8 connection(s)
I20260810 19:27:04.461802 4959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:27:04.469319 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.464206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:36245 (local address 127.19.68.62:40875)
0810 19:27:04.464397 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:04.464402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:04.465783 (+ 1381us) server_negotiation.cc:407] Connection header received
0810 19:27:04.467465 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:04.467469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:04.467522 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:04.467609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:04.468716 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:04.468720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:04.468723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:04.468740 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:04.468772 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:04.468784 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:04.468876 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:04.469001 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 19:27:04.469051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:27:04.470202 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.464708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40875 (local address 127.19.68.2:36245)
0810 19:27:04.465643 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:04.465679 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:04.467185 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:04.467786 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:04.467793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:04.467882 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:04.468445 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:04.468447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:04.468481 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:04.468560 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:04.468858 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:04.468864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:04.468868 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:27:04.469064 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 19:27:04.469297 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":296,"threads_started":1}
I20260810 19:27:04.471441 5219 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40875
I20260810 19:27:04.471730 5219 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:04.471999 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 4959
I20260810 19:27:04.472160 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:27:04.472325 5219 heartbeater.cc:511] Master 127.19.68.62:40875 requested a full tablet report, sending...
I20260810 19:27:04.472615 19728 external_mini_cluster.cc:1428] Setting key c95e02553a6101cd9fd45b46ee727dc4
I20260810 19:27:04.473811 4636 ts_manager.cc:194] Registered new tserver with Master: c828c390460e434188a75e4e003ae5f1 (127.19.68.2:39061)
I20260810 19:27:04.475410 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--builtin_ntp_servers=127.19.68.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 19:27:04.800015 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:04.800258 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:04.800343 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:04.807374 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:04.807468 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:04.807571 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:27:04.816128 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:04.817248 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:04.818928 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:04.831944 5246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:04.831890 5249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:04.831987 5247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:04.832839 5229 server_base.cc:1034] running on GCE node
I20260810 19:27:04.833496 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:04.834453 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:04.835672 5229 hybrid_clock.cc:648] HybridClock initialized: now 1786390024835609 us; error 46 us; skew 500 ppm
I20260810 19:27:04.836009 5229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:27:04.838431 5229 webserver.cc:460] Webserver started at http://127.19.68.3:46867/ using document root <none> and password file <none>
I20260810 19:27:04.839113 5229 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:04.839228 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:04.839561 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:04.842504 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "531e9a2f33ff4440bd4568972bef99ad"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "62fbab698099c3420b194eb93572a4e6"
server_key_iv: "07322b53e091387adee6078aed3cd0d9"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.843338 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "531e9a2f33ff4440bd4568972bef99ad"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "62fbab698099c3420b194eb93572a4e6"
server_key_iv: "07322b53e091387adee6078aed3cd0d9"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.848675 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:27:04.852212 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.853540 5229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 19:27:04.853670 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "531e9a2f33ff4440bd4568972bef99ad"
format_stamp: "Formatted at 2026-08-10 19:27:04 on dist-test-slave-2j7r"
server_key: "62fbab698099c3420b194eb93572a4e6"
server_key_iv: "07322b53e091387adee6078aed3cd0d9"
server_key_version: "encryptionkey@0"
I20260810 19:27:04.853847 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:04.859284 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:04.859994 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:04.860230 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:04.861071 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:04.863298 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:04.863399 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.863503 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:04.863570 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:04.903311 5229 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:35427
I20260810 19:27:04.903369 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:35427 every 8 connection(s)
I20260810 19:27:04.904608 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:27:04.912498 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.907251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:36975 (local address 127.19.68.62:40875)
0810 19:27:04.907430 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:04.907435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:04.908723 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 19:27:04.910411 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:04.910417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:04.910481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:04.910569 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:04.911764 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:04.911769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:04.911772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:04.911789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:04.911821 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:04.911832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:04.911945 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:04.912134 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 19:27:04.912206 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:27:04.913363 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.907679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:40875 (local address 127.19.68.3:36975)
0810 19:27:04.908558 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:04.908601 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:04.910131 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:04.910795 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:04.910803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:04.910897 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:04.911475 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:27:04.911477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:04.911513 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:27:04.911590 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:04.911936 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:04.911947 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:04.911954 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:27:04.912197 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 19:27:04.912426 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260810 19:27:04.914324 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 5229
I20260810 19:27:04.914525 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:27:04.914763 5479 heartbeater.cc:347] Connected to a master server at 127.19.68.62:40875
I20260810 19:27:04.915024 19728 external_mini_cluster.cc:1428] Setting key 48d18143aab3e968213364931f588ecc
I20260810 19:27:04.915031 5479 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:04.915566 5479 heartbeater.cc:511] Master 127.19.68.62:40875 requested a full tablet report, sending...
I20260810 19:27:04.917292 4636 ts_manager.cc:194] Registered new tserver with Master: 531e9a2f33ff4440bd4568972bef99ad (127.19.68.3:35427)
I20260810 19:27:04.930055 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:27:04.936453 4956 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:27:04.934942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.19.68.62:40875)
0810 19:27:04.935093 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:04.935097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:04.935565 (+ 468us) server_negotiation.cc:407] Connection header received
0810 19:27:04.935740 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:04.935744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:04.935797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:04.935878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:04.936247 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45776: BlockingRecv error: recv got EOF from 127.0.0.1:45776 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:27:04.937495 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 4736
I20260810 19:27:04.940771 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:04.938501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.19.68.1:40873)
0810 19:27:04.939030 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:04.939037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:04.939062 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:27:04.939118 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:04.939124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:04.939317 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:04.939434 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:04.940213 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:04.940222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:04.940233 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:04.940253 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:04.940293 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:04.940307 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:04.940406 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:04.940453 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 19:27:04.940529 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260810 19:27:04.942013 4942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:05.024782 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 4736
I20260810 19:27:05.043666 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 4959
I20260810 19:27:05.047084 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:05.044670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.19.68.2:39061)
0810 19:27:05.045273 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:05.045281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:05.045308 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:27:05.045370 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:05.045377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:05.045593 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:05.045713 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:05.046516 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:05.046522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:05.046534 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:05.046545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:05.046589 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:05.046607 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:05.046709 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:05.046758 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 19:27:05.046854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
I20260810 19:27:05.048264 5210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:05.127547 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 4959
I20260810 19:27:05.150646 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 5229
I20260810 19:27:05.154556 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:05.151684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.19.68.3:35427)
0810 19:27:05.152366 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:05.152373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:05.152414 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:27:05.152477 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:05.152483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:05.152691 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:05.152812 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:05.153863 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:05.153869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:05.153880 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:27:05.153898 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:05.153939 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:27:05.153955 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:05.154087 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:05.154139 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 19:27:05.154220 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":349,"threads_started":1}
I20260810 19:27:05.155791 5474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:05.237540 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 5229
I20260810 19:27:05.261165 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 4567
I20260810 19:27:05.262439 4687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:05.363190 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 4567
2026-08-10T19:27:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2356 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:27:05 dist-test-slave-2j7r krb5kdc[5514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:27:05 dist-test-slave-2j7r krb5kdc[5514](info): set up 2 sockets
Aug 10 19:27:05 dist-test-slave-2j7r krb5kdc[5514](info): commencing operation
krb5kdc: starting...
W20260810 19:27:06.651602 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:27:06 dist-test-slave-2j7r krb5kdc[5514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:27:06.926774 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33347
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:33347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:27:07.253620 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:07.253921 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:07.254056 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:07.261183 5533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:27:07.261303 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:07.261382 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:07.261461 5533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:27:07.261523 5533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:27:07.271211 5533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:33347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:07.272524 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:07.273874 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:07.285058 5540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:07.285109 5541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:07.285579 5547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:07.285840 5533 server_base.cc:1034] running on GCE node
I20260810 19:27:07.286604 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:07.287911 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:07.289165 5533 hybrid_clock.cc:648] HybridClock initialized: now 1786390027289096 us; error 40 us; skew 500 ppm
I20260810 19:27:07.289492 5533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:07 dist-test-slave-2j7r krb5kdc[5514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:07.296226 5533 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:27:07.297787 5533 webserver.cc:460] Webserver started at http://127.19.68.62:45285/ using document root <none> and password file <none>
I20260810 19:27:07.298446 5533 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:07.298568 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:07.298867 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:07.301857 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "1bf507c392aa43e4b9bd49a7ea8636bc"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "96ccd6565ff15118e1ebad4a064914af"
server_key_iv: "d8c9337e7c544c613915b4b18073d890"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.302597 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "1bf507c392aa43e4b9bd49a7ea8636bc"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "96ccd6565ff15118e1ebad4a064914af"
server_key_iv: "d8c9337e7c544c613915b4b18073d890"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.307590 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:27:07.311445 5557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:07.312671 5533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:27:07.312834 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "1bf507c392aa43e4b9bd49a7ea8636bc"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "96ccd6565ff15118e1ebad4a064914af"
server_key_iv: "d8c9337e7c544c613915b4b18073d890"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.313014 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:07.327754 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:07.331485 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:07.331745 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:07.348708 5533 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:33347
I20260810 19:27:07.348752 5673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:33347 every 8 connection(s)
I20260810 19:27:07.350011 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:27:07.353521 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:07.354436 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 5533
I20260810 19:27:07.354676 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:27:07.355547 19728 external_mini_cluster.cc:1428] Setting key bce6fc7c75db7b32cbc187602c633e85
I20260810 19:27:07.367305 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: Bootstrap starting.
Aug 10 19:27:07 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:07.370779 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:07.371817 5675 log.cc:824] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:07.374714 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: No bootstrap required, opened a new log
I20260810 19:27:07.378089 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:07.357641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.19.68.62:33347)
0810 19:27:07.358383 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:07.358394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:07.358438 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:27:07.359992 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:07.360012 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:07.360371 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:07.360668 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:07.362510 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.363781 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.365629 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.366281 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.366415 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:07.370912 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:07.370921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:07.370934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:07.370970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:07.374074 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:07.374740 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:07.374747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:07.374752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:07.374941 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:07.375316 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:07.375320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:07.375322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:07.375858 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:07.376187 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:07.376629 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 19:27:07.377044 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":334,"threads_started":1}
I20260810 19:27:07.382423 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } }
I20260810 19:27:07.382761 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:07.382866 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf507c392aa43e4b9bd49a7ea8636bc, State: Initialized, Role: FOLLOWER
I20260810 19:27:07.383380 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } }
I20260810 19:27:07.383579 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:27:07.383701 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:27:07.383843 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:07.386066 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } }
I20260810 19:27:07.386598 5675 leader_election.cc:302] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bf507c392aa43e4b9bd49a7ea8636bc; no voters:
I20260810 19:27:07.387198 5675 leader_election.cc:288] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:27:07.387326 5684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:07.388000 5684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [term 1 LEADER]: Becoming Leader. State: Replica: 1bf507c392aa43e4b9bd49a7ea8636bc, State: Running, Role: LEADER
I20260810 19:27:07.388525 5684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } }
I20260810 19:27:07.389051 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:27:07.391264 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf507c392aa43e4b9bd49a7ea8636bc. Latest consensus state: current_term: 1 leader_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } } }
I20260810 19:27:07.391639 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:07.391469 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf507c392aa43e4b9bd49a7ea8636bc" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33347 } } }
I20260810 19:27:07.391935 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:07.393940 5701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:27:07.398904 5701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:27:07.407192 5701 catalog_manager.cc:1347] Generated new cluster ID: ec006766c6784ed38808e4efa7e8b585
I20260810 19:27:07.407307 5701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:27:07.441797 5701 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:27:07.443708 5701 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:27:07.452572 5701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: Generated new TSK 0
I20260810 19:27:07.453621 5701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:27:07.522989 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:27:07.852237 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:07.852512 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:07.852665 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:07.859599 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:07.859721 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:07.859871 5731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:27:07.869149 5731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:07.870299 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:07.871552 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:07.884706 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:07.884645 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:07.884676 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:07.890086 5731 server_base.cc:1034] running on GCE node
I20260810 19:27:07.890704 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:07.891587 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:07.892820 5731 hybrid_clock.cc:648] HybridClock initialized: now 1786390027892761 us; error 38 us; skew 500 ppm
I20260810 19:27:07.893177 5731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:07 dist-test-slave-2j7r krb5kdc[5514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:07.896416 5731 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:27:07.897949 5731 webserver.cc:460] Webserver started at http://127.19.68.1:38963/ using document root <none> and password file <none>
I20260810 19:27:07.898638 5731 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:07.898739 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:07.898990 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:07.902001 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "b336b49aa6574b50a1b5362e26618c6e"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "82ad5a7e2ecb45886bc0740f901d2e23"
server_key_iv: "a8716c85674a49a5c0ead2d2442d6075"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.902750 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "b336b49aa6574b50a1b5362e26618c6e"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "82ad5a7e2ecb45886bc0740f901d2e23"
server_key_iv: "a8716c85674a49a5c0ead2d2442d6075"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.908047 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:27:07.911604 5756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:07.913062 5731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:27:07.913276 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "b336b49aa6574b50a1b5362e26618c6e"
format_stamp: "Formatted at 2026-08-10 19:27:07 on dist-test-slave-2j7r"
server_key: "82ad5a7e2ecb45886bc0740f901d2e23"
server_key_iv: "a8716c85674a49a5c0ead2d2442d6075"
server_key_version: "encryptionkey@0"
I20260810 19:27:07.913472 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:07.930902 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:07.934789 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:07.935061 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:07.936028 5731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:07.937525 5731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:07.937601 5731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:07.937705 5731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:07.937764 5731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:07.967705 5731 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:38489
I20260810 19:27:07.967749 5980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:38489 every 8 connection(s)
I20260810 19:27:07.968950 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:27:07.969930 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 5731
I20260810 19:27:07.970163 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:27:07.971014 19728 external_mini_cluster.cc:1428] Setting key a887705404e16fa241ea5e25ba370409
Aug 10 19:27:07 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:07.990429 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:07.971761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:51787 (local address 127.19.68.62:33347)
0810 19:27:07.972260 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:07.972266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:07.973366 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 19:27:07.975721 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:07.975726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:07.975793 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:07.975885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:07.977745 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.978460 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.980475 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.980852 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.980947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:07.984790 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:07.984796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:07.984799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:07.984832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:07.987255 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:07.988083 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:07.988087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:07.988090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:07.988215 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:07.988651 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:07.988654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:07.988656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:07.988925 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:07.989065 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:07.989860 (+ 795us) server_negotiation.cc:299] Negotiation successful
0810 19:27:07.990096 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260810 19:27:07.991168 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:07.972382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33347 (local address 127.19.68.1:51787)
0810 19:27:07.973189 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:07.973248 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:07.975431 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:07.976138 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:07.976149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:07.976827 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:07.977592 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:07.977616 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.978610 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.978616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:07.980338 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:07.980353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:07.981072 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:07.981079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:07.981354 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:07.982032 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:07.982065 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:07.984574 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:07.987396 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:07.987404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:07.987419 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:07.987947 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:07.988316 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:07.988320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:07.988324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:07.988547 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:07.989065 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:07.989073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:07.989446 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 19:27:07.989705 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 19:27:07.990098 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260810 19:27:07.992781 5981 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33347
I20260810 19:27:07.993108 5981 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:07.993870 5981 heartbeater.cc:511] Master 127.19.68.62:33347 requested a full tablet report, sending...
I20260810 19:27:07.995908 5601 ts_manager.cc:194] Registered new tserver with Master: b336b49aa6574b50a1b5362e26618c6e (127.19.68.1:38489)
I20260810 19:27:07.997368 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:51787
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:27:08.035683 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:27:08.358932 5993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:08.359198 5993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:08.359337 5993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:08.366360 5993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:08.366467 5993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:08.366621 5993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:27:08.375932 5993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:08.377153 5993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:08.378373 5993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:08.390875 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:08.390852 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:08.391075 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:08.392369 5993 server_base.cc:1034] running on GCE node
I20260810 19:27:08.393019 5993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:08.393994 5993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:08.395201 5993 hybrid_clock.cc:648] HybridClock initialized: now 1786390028395160 us; error 40 us; skew 500 ppm
I20260810 19:27:08.395478 5993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:08 dist-test-slave-2j7r krb5kdc[5514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390028, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:08.398470 5993 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:27:08.399830 5993 webserver.cc:460] Webserver started at http://127.19.68.2:41593/ using document root <none> and password file <none>
I20260810 19:27:08.400517 5993 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:08.400619 5993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:08.400943 5993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:08.403793 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "35d700a98618495ba4fc9fb07595a012"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "b3fbc41a43badcef437567b636906b8e"
server_key_iv: "45fa31a0f332e03b38c1be451ce8d9df"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.404518 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "35d700a98618495ba4fc9fb07595a012"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "b3fbc41a43badcef437567b636906b8e"
server_key_iv: "45fa31a0f332e03b38c1be451ce8d9df"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.409526 5993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:27:08.412818 6018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:08.414095 5993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:27:08.414227 5993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "35d700a98618495ba4fc9fb07595a012"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "b3fbc41a43badcef437567b636906b8e"
server_key_iv: "45fa31a0f332e03b38c1be451ce8d9df"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.414394 5993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:08.459471 5993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:08.463568 5993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:08.463850 5993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:08.464805 5993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:08.466215 5993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:08.466306 5993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:08.466429 5993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:08.466485 5993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:08.495610 5993 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:42321
I20260810 19:27:08.495661 6233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:42321 every 8 connection(s)
I20260810 19:27:08.496855 5993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:27:08.502592 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 5993
I20260810 19:27:08.502787 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:27:08.503356 19728 external_mini_cluster.cc:1428] Setting key 99d1ee306990f6c5695f4d9c1cba41a4
Aug 10 19:27:08 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390028, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:08.518574 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:08.499358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:44609 (local address 127.19.68.62:33347)
0810 19:27:08.499801 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:08.499806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:08.500663 (+ 857us) server_negotiation.cc:407] Connection header received
0810 19:27:08.502525 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:08.502529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:08.502600 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:08.502706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:08.504602 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:08.505702 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:08.507853 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:08.508264 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:08.508362 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:08.512293 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:08.512299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:08.512303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:08.512341 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:08.515275 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:08.516062 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:08.516067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:08.516070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:08.516223 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:08.516691 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:08.516695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:08.516697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:08.517005 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:08.517168 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:08.517919 (+ 751us) server_negotiation.cc:299] Negotiation successful
0810 19:27:08.518191 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260810 19:27:08.519636 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:08.499824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33347 (local address 127.19.68.2:44609)
0810 19:27:08.500506 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:08.500543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:08.502197 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:08.502979 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:08.502991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:08.503699 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:08.504433 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:08.504458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:08.505882 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:08.505888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:08.507694 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:08.507708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:08.508494 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:08.508501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:08.508749 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:08.509454 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:08.509488 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:08.512031 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:08.515406 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:08.515414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:08.515429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:08.515955 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:08.516321 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:08.516325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:08.516328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:08.516566 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:08.517152 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:08.517160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:08.517475 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 19:27:08.517797 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 19:27:08.518336 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260810 19:27:08.520994 6234 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33347
I20260810 19:27:08.521329 6234 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:08.522205 6234 heartbeater.cc:511] Master 127.19.68.62:33347 requested a full tablet report, sending...
I20260810 19:27:08.523675 5601 ts_manager.cc:194] Registered new tserver with Master: 35d700a98618495ba4fc9fb07595a012 (127.19.68.2:42321)
I20260810 19:27:08.524472 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:44609
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:27:08.566605 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:27:08.902335 6248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:08.902611 6248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:08.902710 6248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:08.910005 6248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:08.910130 6248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:08.910300 6248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:27:08.920729 6248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:08.922048 6248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:08.923417 6248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:08.936861 6258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:08.936839 6266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:08.936833 6260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:08.937654 6248 server_base.cc:1034] running on GCE node
I20260810 19:27:08.938313 6248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:08.939323 6248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:08.940557 6248 hybrid_clock.cc:648] HybridClock initialized: now 1786390028940493 us; error 53 us; skew 500 ppm
I20260810 19:27:08.940923 6248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:08 dist-test-slave-2j7r krb5kdc[5514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390028, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:08.944136 6248 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:27:08.945588 6248 webserver.cc:460] Webserver started at http://127.19.68.3:43975/ using document root <none> and password file <none>
I20260810 19:27:08.946278 6248 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:08.946395 6248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:08.946754 6248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:27:08.950040 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "4619843c1c904d2497da6471f777745b"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "4d671f4080014e24d8c7a9d9324ac97c"
server_key_iv: "3647ac3d1f82dd589d6b8eff23cd4afa"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.950820 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "4619843c1c904d2497da6471f777745b"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "4d671f4080014e24d8c7a9d9324ac97c"
server_key_iv: "3647ac3d1f82dd589d6b8eff23cd4afa"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.955781 6248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 19:27:08.959141 6282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:08.960407 6248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:27:08.960556 6248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "4619843c1c904d2497da6471f777745b"
format_stamp: "Formatted at 2026-08-10 19:27:08 on dist-test-slave-2j7r"
server_key: "4d671f4080014e24d8c7a9d9324ac97c"
server_key_iv: "3647ac3d1f82dd589d6b8eff23cd4afa"
server_key_version: "encryptionkey@0"
I20260810 19:27:08.960728 6248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:08.979095 6248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:08.983199 6248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:08.983492 6248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:08.984452 6248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:08.985844 6248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:08.985922 6248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:08.986059 6248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:08.986114 6248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:09.000216 5981 heartbeater.cc:503] Master 127.19.68.62:33347 was elected leader, sending a full tablet report...
I20260810 19:27:09.018162 6248 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:39583
I20260810 19:27:09.018209 6506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:39583 every 8 connection(s)
I20260810 19:27:09.019539 6248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:27:09.023308 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 6248
I20260810 19:27:09.023538 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:27:09.024075 19728 external_mini_cluster.cc:1428] Setting key 674d356aaa2b640ef2ed83f31860e356
Aug 10 19:27:09 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390028, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:09.038877 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.022236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:51181 (local address 127.19.68.62:33347)
0810 19:27:09.022664 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.022669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.023768 (+ 1099us) server_negotiation.cc:407] Connection header received
0810 19:27:09.025733 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.025739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.025829 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.025924 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:09.027390 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.028151 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.029728 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.030115 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.030221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.033436 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:09.033440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:09.033444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:09.033476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:09.035835 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.036520 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.036524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.036526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.036651 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.037065 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.037068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.037071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.037449 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:09.037598 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.038233 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.038450 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":209,"threads_started":1}
I20260810 19:27:09.039744 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.022771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:33347 (local address 127.19.68.3:51181)
0810 19:27:09.023610 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.023668 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.025490 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.026196 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.026204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.026699 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:09.027231 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.027247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.028300 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.028304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.029589 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.029602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.030330 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.030337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.030560 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.031116 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:09.031142 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:09.033236 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:09.035973 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:09.035982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:09.035996 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:09.036379 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:09.036757 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:09.036762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:09.036766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:09.036951 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:09.037577 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:09.037584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:09.037888 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.038115 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.038603 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":361,"threads_started":1}
I20260810 19:27:09.041052 6508 heartbeater.cc:347] Connected to a master server at 127.19.68.62:33347
I20260810 19:27:09.041383 6508 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:09.042117 6508 heartbeater.cc:511] Master 127.19.68.62:33347 requested a full tablet report, sending...
I20260810 19:27:09.043538 5601 ts_manager.cc:194] Registered new tserver with Master: 4619843c1c904d2497da6471f777745b (127.19.68.3:39583)
I20260810 19:27:09.044394 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:51181
I20260810 19:27:09.051270 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:27:09.067941 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.055600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.19.68.62:33347)
0810 19:27:09.055764 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.055768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.056264 (+ 496us) server_negotiation.cc:407] Connection header received
0810 19:27:09.056401 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.056406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.056464 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.056539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:09.057951 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.058646 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.059835 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.060244 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.060331 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.062828 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:09.062832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:09.062835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:09.062866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:09.065539 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.066132 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.066136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.066139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.066300 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.066623 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.066626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.066628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.066931 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:09.067116 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.067544 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.067709 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:27:09.072024 5601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:27:09.079169 5601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:27:09.100955 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.091210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:42321 (local address 127.0.0.1:37500)
0810 19:27:09.091919 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.091962 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.092158 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.093483 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.093490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.093530 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:09.094141 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.094157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.096042 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.096047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.098167 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.098183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.099916 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.099924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.100126 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.100185 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.100284 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.100466 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":128,"spinlock_wait_cycles":23808,"thread_start_us":861,"threads_started":1}
I20260810 19:27:09.101465 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.091529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36934)
0810 19:27:09.092259 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.092276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.092387 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.093525 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.093529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.093574 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:09.094161 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.094176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.096216 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.096225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.098861 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.098872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.100579 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.100585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.100797 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.100825 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.100906 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.101029 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":448,"spinlock_wait_cycles":8960,"thread_start_us":577,"threads_started":1}
I20260810 19:27:09.101511 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.091867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.19.68.1:38489)
0810 19:27:09.092919 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.092927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.092954 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:27:09.093057 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.093065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.093313 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.093788 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:09.094413 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.096079 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.099037 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.100346 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.100466 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.100697 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.100981 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.101176 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":1512,"threads_started":1}
I20260810 19:27:09.102665 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.091905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.19.68.2:42321)
0810 19:27:09.092778 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.092787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.092817 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:27:09.092913 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.092921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.093212 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.094097 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:09.094334 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.095866 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.098311 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.099676 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.101880 (+ 2204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.102100 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.102167 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.102338 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":1857,"threads_started":1}
I20260810 19:27:09.103262 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.093484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.19.68.3:39583)
0810 19:27:09.094765 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.094773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.095294 (+ 521us) server_negotiation.cc:407] Connection header received
0810 19:27:09.095429 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.095434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.095593 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.095769 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:09.096848 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.098440 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.100847 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.102197 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.102319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.102535 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.102778 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.102929 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":949,"threads_started":1}
I20260810 19:27:09.103515 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.092838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:39583 (local address 127.0.0.1:38772)
0810 19:27:09.095179 (+ 2341us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.095192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.095303 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.096034 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.096038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.096060 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:09.096664 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.096676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.098602 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.098607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.100655 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.100671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.102405 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.102411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.102574 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.102605 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.102702 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.102827 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2245,"mutex_wait_us":97,"thread_start_us":2106,"threads_started":1}
I20260810 19:27:09.105537 5851 tablet_service.cc:1467] Processing CreateTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 (DEFAULT_TABLE table=test-table [id=aece66f37dbe41e3b962876b0ce867bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:09.106160 6108 tablet_service.cc:1467] Processing CreateTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 (DEFAULT_TABLE table=test-table [id=aece66f37dbe41e3b962876b0ce867bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:09.107153 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab81a79ca1b14b3b8f8598dc341e7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:09.107234 6108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab81a79ca1b14b3b8f8598dc341e7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:09.107069 6383 tablet_service.cc:1467] Processing CreateTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 (DEFAULT_TABLE table=test-table [id=aece66f37dbe41e3b962876b0ce867bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:09.108431 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab81a79ca1b14b3b8f8598dc341e7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:09.118633 6542 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Bootstrap starting.
I20260810 19:27:09.121311 6544 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Bootstrap starting.
I20260810 19:27:09.122658 6542 tablet_bootstrap.cc:654] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:09.122862 6545 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Bootstrap starting.
I20260810 19:27:09.124603 6542 log.cc:824] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:09.125563 6544 tablet_bootstrap.cc:654] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:09.125834 6545 tablet_bootstrap.cc:654] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:09.126746 6544 log.cc:824] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:09.126818 6545 log.cc:824] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:09.127967 6542 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: No bootstrap required, opened a new log
I20260810 19:27:09.128270 6542 ts_tablet_manager.cc:1397] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 19:27:09.129311 6545 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: No bootstrap required, opened a new log
I20260810 19:27:09.129554 6545 ts_tablet_manager.cc:1397] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 19:27:09.136989 6545 raft_consensus.cc:348] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.137627 6545 raft_consensus.cc:374] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:09.137748 6545 raft_consensus.cc:729] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4619843c1c904d2497da6471f777745b, State: Initialized, Role: FOLLOWER
I20260810 19:27:09.138448 6545 consensus_queue.cc:260] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.138998 6542 raft_consensus.cc:348] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.139482 6542 raft_consensus.cc:374] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:09.139607 6508 heartbeater.cc:503] Master 127.19.68.62:33347 was elected leader, sending a full tablet report...
I20260810 19:27:09.139634 6542 raft_consensus.cc:729] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d700a98618495ba4fc9fb07595a012, State: Initialized, Role: FOLLOWER
I20260810 19:27:09.140066 6544 tablet_bootstrap.cc:492] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: No bootstrap required, opened a new log
I20260810 19:27:09.140266 6545 ts_tablet_manager.cc:1428] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 19:27:09.140364 6544 ts_tablet_manager.cc:1397] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20260810 19:27:09.140254 6542 consensus_queue.cc:260] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.146777 6234 heartbeater.cc:503] Master 127.19.68.62:33347 was elected leader, sending a full tablet report...
I20260810 19:27:09.147239 6542 ts_tablet_manager.cc:1428] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260810 19:27:09.148900 6544 raft_consensus.cc:348] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.149371 6544 raft_consensus.cc:374] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:09.149480 6544 raft_consensus.cc:729] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b336b49aa6574b50a1b5362e26618c6e, State: Initialized, Role: FOLLOWER
I20260810 19:27:09.150058 6544 consensus_queue.cc:260] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.151535 6544 ts_tablet_manager.cc:1428] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
W20260810 19:27:09.221313 5982 tablet.cc:2369] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:27:09.248975 6236 tablet.cc:2369] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:27:09.253120 6551 raft_consensus.cc:482] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:27:09.253417 6551 raft_consensus.cc:504] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.255167 6551 leader_election.cc:288] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35d700a98618495ba4fc9fb07595a012 (127.19.68.2:42321), b336b49aa6574b50a1b5362e26618c6e (127.19.68.1:38489)
I20260810 19:27:09.260865 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.255581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:42321 (local address 127.19.68.3:38841)
0810 19:27:09.255940 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.255955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.256088 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.256421 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.256426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.256453 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:09.256671 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.256680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.257720 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.257726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.259009 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.259022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.260116 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.260122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.260304 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.260329 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.260408 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.260537 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"spinlock_wait_cycles":18816}
I20260810 19:27:09.260856 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.255581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:38841 (local address 127.19.68.2:42321)
0810 19:27:09.255788 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.255794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.256041 (+ 247us) server_negotiation.cc:407] Connection header received
0810 19:27:09.256229 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.256233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.256296 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.256369 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:09.256822 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.257571 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.259162 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.259953 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.260074 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.260169 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.260456 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.260584 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:27:09.261400 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.255637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.19.68.3:60615)
0810 19:27:09.256396 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.256412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:09.256551 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:09.256845 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:09.256849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:09.256872 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:09.257244 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.257254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.258237 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.258242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.259558 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:09.259570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.260672 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.260678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:09.260834 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.260860 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:27:09.260943 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:27:09.261070 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":419,"threads_started":1}
I20260810 19:27:09.261655 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.255660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:60615 (local address 127.19.68.1:38489)
0810 19:27:09.255852 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.255857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.256504 (+ 647us) server_negotiation.cc:407] Connection header received
0810 19:27:09.256682 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.256686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.256746 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.256818 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:09.257416 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.258095 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.259710 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.260476 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.260591 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.260686 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.261293 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.261436 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:27:09.261906 6150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab81a79ca1b14b3b8f8598dc341e7848" candidate_uuid: "4619843c1c904d2497da6471f777745b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d700a98618495ba4fc9fb07595a012" is_pre_election: true
I20260810 19:27:09.262486 6150 raft_consensus.cc:2454] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4619843c1c904d2497da6471f777745b in term 0.
I20260810 19:27:09.262543 5886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab81a79ca1b14b3b8f8598dc341e7848" candidate_uuid: "4619843c1c904d2497da6471f777745b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b336b49aa6574b50a1b5362e26618c6e" is_pre_election: true
I20260810 19:27:09.263067 5886 raft_consensus.cc:2454] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4619843c1c904d2497da6471f777745b in term 0.
I20260810 19:27:09.263291 6287 leader_election.cc:302] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d700a98618495ba4fc9fb07595a012, 4619843c1c904d2497da6471f777745b; no voters:
I20260810 19:27:09.263711 6551 raft_consensus.cc:2781] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:27:09.263844 6551 raft_consensus.cc:482] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:27:09.263938 6551 raft_consensus.cc:3037] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:09.266278 6551 raft_consensus.cc:504] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.266883 6551 leader_election.cc:288] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [CANDIDATE]: Term 1 election: Requested vote from peers 35d700a98618495ba4fc9fb07595a012 (127.19.68.2:42321), b336b49aa6574b50a1b5362e26618c6e (127.19.68.1:38489)
I20260810 19:27:09.267277 6150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab81a79ca1b14b3b8f8598dc341e7848" candidate_uuid: "4619843c1c904d2497da6471f777745b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d700a98618495ba4fc9fb07595a012"
I20260810 19:27:09.267355 5886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab81a79ca1b14b3b8f8598dc341e7848" candidate_uuid: "4619843c1c904d2497da6471f777745b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b336b49aa6574b50a1b5362e26618c6e"
I20260810 19:27:09.267454 6150 raft_consensus.cc:3037] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:09.267553 5886 raft_consensus.cc:3037] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:09.269740 6150 raft_consensus.cc:2454] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4619843c1c904d2497da6471f777745b in term 1.
I20260810 19:27:09.269764 5886 raft_consensus.cc:2454] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4619843c1c904d2497da6471f777745b in term 1.
I20260810 19:27:09.270136 6288 leader_election.cc:302] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4619843c1c904d2497da6471f777745b, b336b49aa6574b50a1b5362e26618c6e; no voters:
I20260810 19:27:09.270440 6551 raft_consensus.cc:2781] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:09.270916 6551 raft_consensus.cc:686] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 1 LEADER]: Becoming Leader. State: Replica: 4619843c1c904d2497da6471f777745b, State: Running, Role: LEADER
W20260810 19:27:09.271039 6509 tablet.cc:2369] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:27:09.271481 6551 consensus_queue.cc:237] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } }
I20260810 19:27:09.276283 5601 catalog_manager.cc:5495] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4619843c1c904d2497da6471f777745b (127.19.68.3). New cstate: current_term: 1 leader_uuid: "4619843c1c904d2497da6471f777745b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4619843c1c904d2497da6471f777745b" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39583 } health_report { overall_health: HEALTHY } } }
I20260810 19:27:09.294397 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.289544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.19.68.3:39583)
0810 19:27:09.289714 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.289718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.289841 (+ 123us) server_negotiation.cc:407] Connection header received
0810 19:27:09.290009 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.290012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.290102 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.290222 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:27:09.290495 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.291209 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.292438 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.292888 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.292977 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.293411 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:27:09.293956 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:27:09.294063 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.294130 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.294195 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:27:09.302278 6150 raft_consensus.cc:1261] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 1 FOLLOWER]: Refusing update from remote peer 4619843c1c904d2497da6471f777745b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:27:09.302280 5886 raft_consensus.cc:1261] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 1 FOLLOWER]: Refusing update from remote peer 4619843c1c904d2497da6471f777745b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:27:09.303153 6564 consensus_queue.cc:1035] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d700a98618495ba4fc9fb07595a012" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:27:09.303503 6551 consensus_queue.cc:1035] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b336b49aa6574b50a1b5362e26618c6e" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:27:09.328213 5599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:27:09.328490 5599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:27:09.331972 5599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bf507c392aa43e4b9bd49a7ea8636bc: Sending DeleteTablet for 3 replicas of tablet ab81a79ca1b14b3b8f8598dc341e7848
I20260810 19:27:09.333221 6108 tablet_service.cc:1514] Processing DeleteTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:37500
I20260810 19:27:09.333439 6383 tablet_service.cc:1514] Processing DeleteTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:38772
I20260810 19:27:09.333467 5851 tablet_service.cc:1514] Processing DeleteTablet for tablet ab81a79ca1b14b3b8f8598dc341e7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:09 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:36934
I20260810 19:27:09.334965 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 5731
I20260810 19:27:09.341619 6592 tablet_replica.cc:330] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: stopping tablet replica
I20260810 19:27:09.342255 6592 raft_consensus.cc:2229] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:27:09.342872 6592 raft_consensus.cc:2258] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:09.343721 6590 tablet_replica.cc:330] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: stopping tablet replica
I20260810 19:27:09.344164 6590 raft_consensus.cc:2229] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:27:09.344683 6590 raft_consensus.cc:2258] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:09.345479 6592 ts_tablet_manager.cc:1905] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:27:09.346956 6590 ts_tablet_manager.cc:1905] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:27:09.356531 6592 ts_tablet_manager.cc:1918] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:09.357607 6591 tablet_replica.cc:330] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: stopping tablet replica
I20260810 19:27:09.358099 6590 ts_tablet_manager.cc:1918] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:09.358089 6591 raft_consensus.cc:2229] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:27:09.358368 6590 log.cc:1196] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/ab81a79ca1b14b3b8f8598dc341e7848
I20260810 19:27:09.358788 6591 raft_consensus.cc:2258] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:09.359058 6590 ts_tablet_manager.cc:1939] T ab81a79ca1b14b3b8f8598dc341e7848 P 35d700a98618495ba4fc9fb07595a012: Deleting consensus metadata
Aug 10 19:27:09 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:27:09.361137 6591 ts_tablet_manager.cc:1905] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:27:09.361933 6592 log.cc:1196] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/ab81a79ca1b14b3b8f8598dc341e7848
I20260810 19:27:09.368103 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.336275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.19.68.1:38489)
0810 19:27:09.336467 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.336474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.345529 (+ 9055us) server_negotiation.cc:407] Connection header received
0810 19:27:09.345814 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.345819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.345896 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.345994 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:09.347638 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.348705 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.350548 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.351180 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.351263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.361002 (+ 9739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:09.361017 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:09.361033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:09.361081 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:09.364677 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.365590 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.365598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.365601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.365761 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.366225 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.366230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.366234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.366688 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:09.366969 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.367514 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.367735 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:27:09.369278 6591 ts_tablet_manager.cc:1918] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:09.369508 6591 log.cc:1196] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/ab81a79ca1b14b3b8f8598dc341e7848
I20260810 19:27:09.369633 5976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:09.371647 5559 catalog_manager.cc:4841] TS 35d700a98618495ba4fc9fb07595a012 (127.19.68.2:42321): tablet ab81a79ca1b14b3b8f8598dc341e7848 (table test-table [id=aece66f37dbe41e3b962876b0ce867bf]) successfully deleted
I20260810 19:27:09.372426 6591 ts_tablet_manager.cc:1939] T ab81a79ca1b14b3b8f8598dc341e7848 P 4619843c1c904d2497da6471f777745b: Deleting consensus metadata
I20260810 19:27:09.374342 5561 catalog_manager.cc:4841] TS 4619843c1c904d2497da6471f777745b (127.19.68.3:39583): tablet ab81a79ca1b14b3b8f8598dc341e7848 (table test-table [id=aece66f37dbe41e3b962876b0ce867bf]) successfully deleted
I20260810 19:27:09.374863 6592 ts_tablet_manager.cc:1939] T ab81a79ca1b14b3b8f8598dc341e7848 P b336b49aa6574b50a1b5362e26618c6e: Deleting consensus metadata
I20260810 19:27:09.459600 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 5731
I20260810 19:27:09.486600 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 5993
W20260810 19:27:09.486680 5561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:38489 (error 108)
I20260810 19:27:09.489004 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.488471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36960)
0810 19:27:09.488665 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.488789 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 19:27:09.489332 5561 catalog_manager.cc:4584] TS b336b49aa6574b50a1b5362e26618c6e (127.19.68.1:38489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab81a79ca1b14b3b8f8598dc341e7848: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Aug 10 19:27:09 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:27:09.504971 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.487451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.19.68.2:42321)
0810 19:27:09.487642 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.487648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.487670 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:27:09.487873 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.487881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.487991 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.488068 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:09.489908 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.491130 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.492370 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.493008 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.493307 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.497961 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:09.497975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:09.497990 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:09.498042 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:09.501869 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.502709 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.502720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.502723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.502879 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.503291 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.503296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.503300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.503785 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:09.504057 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.504354 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.504596 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:27:09.505836 6222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:09.543179 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.542593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36970)
0810 19:27:09.542849 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.542978 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 19:27:09.544382 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.543976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36976)
0810 19:27:09.544132 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.544238 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:27:09.596590 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 5993
I20260810 19:27:09.606747 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.606136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36992)
0810 19:27:09.606396 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.606526 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 19:27:09.608867 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.608527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36994)
0810 19:27:09.608680 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.608751 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:27:09.631119 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 6248
Aug 10 19:27:09 dist-test-slave-2j7r krb5kdc[5514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:27:09.646974 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.632004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.19.68.3:39583)
0810 19:27:09.632217 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:09.632222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:09.632242 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:27:09.632366 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:09.632370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:09.632449 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:09.632530 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:09.633829 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.634626 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.635870 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:09.636322 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:09.636440 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:09.640879 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:09.640894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:09.640917 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:09.640950 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:09.643836 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.644626 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.644636 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.644640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.644853 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:09.645235 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:09.645240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:09.645244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:09.645757 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:09.646089 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:09.646283 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 19:27:09.646514 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 19:27:09.647805 6501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:09.690220 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.689746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:36998)
0810 19:27:09.689944 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.690040 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 19:27:09.691299 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.690900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:37012)
0810 19:27:09.691101 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.691188 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 19:27:09.769829 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 6248
I20260810 19:27:09.797598 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 5533
I20260810 19:27:09.798815 5666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:09.907830 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:09.907210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38489 (local address 127.0.0.1:37014)
0810 19:27:09.907391 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:09.907534 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:27:09.908229 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 5533
2026-08-10T19:27:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4547 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:27:10 dist-test-slave-2j7r krb5kdc[6606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:27:10 dist-test-slave-2j7r krb5kdc[6606](info): set up 2 sockets
Aug 10 19:27:10 dist-test-slave-2j7r krb5kdc[6606](info): commencing operation
krb5kdc: starting...
W20260810 19:27:11.162757 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:27:11 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:27:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangeradmin_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangeradmin_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangerlookup_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangerlookup_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
I20260810 19:27:11.547698 19728 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTNmuTH/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/postgres -l logfile start
2026-08-10 19:27:13.849 UTC [6668] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 19:27:13.849 UTC [6668] LOG: listening on IPv4 address "127.19.68.20", port 39371
2026-08-10 19:27:13.852 UTC [6668] LOG: listening on Unix socket "/tmp/.s.PGSQL.39371"
2026-08-10 19:27:13.876 UTC [6670] LOG: database system was shut down at 2026-08-10 19:27:12 UTC
2026-08-10 19:27:13.879 UTC [6668] LOG: database system is ready to accept connections
I20260810 19:27:15.040830 19728 mini_postgres.cc:96] Postgres bound to 39371
2026-08-10 19:27:15.066 UTC [6685] FATAL: database "slave" does not exist
127.19.68.20:39371 - accepting connections
I20260810 19:27:15.067669 19728 mini_ranger.cc:161] Starting Ranger...
I20260810 19:27:15.100569 19728 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 19:27:15.530958 19728 mini_ranger.cc:87] Created ranger Postgres database
I20260810 19:27:15.531085 19728 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin
2026-08-10 19:27:15,880 [I] DB FLAVOR :POSTGRES
2026-08-10 19:27:15,880 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:27:15,880 [I] Checking connection..
2026-08-10 19:27:15,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 19:27:16,127 [I] Checking connection passed.
2026-08-10 19:27:16,127 [I] --------- Verifying version history table ---------
2026-08-10 19:27:16,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:27:16,386 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:27:16,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:27:16,650 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:27:16,650 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 19:27:16,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 19:27:16.898 UTC [6792] WARNING: there is no transaction in progress
2026-08-10 19:27:16,906 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 19:27:16,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:27:17,150 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 19:27:17,150 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:27:17,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 19:27:17,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 19:27:17,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2j7r.c.gcp-upstream.internal','N') ;"
2026-08-10 19:27:17,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:27:18,165 [I] Table x_portal_user does not exist in database ranger
2026-08-10 19:27:18,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:27:18,404 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 19:27:18,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:27:18,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:27:18,894 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 19:27:18,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 19:27:19.791 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.807 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.822 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.837 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.851 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.868 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.919 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.924 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.938 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.948 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.958 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.971 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.980 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.986 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.991 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:19.997 UTC [6970] WARNING: there is no transaction in progress
2026-08-10 19:27:20,564 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 19:27:20,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 19:27:20,817 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 19:27:20,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:27:21,075 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 19:27:21,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:27:21,335 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 19:27:21,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:27:21,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:27:21,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2j7r.c.gcp-upstream.internal';"
2026-08-10 19:27:22,093 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 19:27:22,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 19:27:22,350 [I] DB_PATCHES have already been applied
I20260810 19:27:22.355834 19728 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTNmuTH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 19:27:22.355922 19728 mini_ranger.cc:193] Using host: 127.19.68.20
I20260810 19:27:22.378981 19728 mini_ranger.cc:227] Ranger admin URL: http://127.19.68.20:33801
Aug 10, 2026 7:27:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:27:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:27:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:27:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:27:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:27:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:27:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 19:27:23.693532 19728 mini_ranger.cc:160] Time spent starting Ranger: real 8.626s user 0.000s sys 0.071s
Aug 10, 2026 7:27:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:27:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:27:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:27:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:27:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:27:45.291462 19728 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangerkms_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/rangerkms_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.68.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.68.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:27:45 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390065, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:27:45.491109 19728 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:27:45.516846 19728 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:27:46.284416 19728 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:27:46.284564 19728 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms
I20260810 19:27:46.284703 19728 mini_ranger_kms.cc:227] Using postgres at 127.19.68.20:39371
2026-08-10 19:27:46,777 [I] DB FLAVOR :POSTGRES
2026-08-10 19:27:46,777 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:27:46,777 [I] Checking connection
2026-08-10 19:27:46,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:27:47,025 [I] connection success
2026-08-10 19:27:47,025 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:27:47,025 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:27:47,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:27:47,267 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:27:47,267 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:27:47,267 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:27:47,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:39371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:27:47,537 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:27:48.017532 19728 mini_ranger_kms.cc:310] Created kms service
I20260810 19:27:48.188680 19728 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:27:48.234182 19728 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:27:48 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390065, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:48.652143 19728 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:27:48.652524 19728 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:27:48.652591 19728 mini_ranger_kms.cc:240] Using host: 127.19.68.20
I20260810 19:27:48.679261 19728 mini_ranger_kms.cc:276] Ranger KMS PID: 7325
I20260810 19:27:48.679476 19728 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.68.20:32857
Aug 10, 2026 7:27:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:27:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:27:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:27:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:27:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:27:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:27:50 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786390070, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 19:27:51 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390070, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:51.362089 19728 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.871s user 0.166s sys 0.369s
I20260810 19:27:51.362489 19728 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:27:51.362735 19728 mini_ranger_kms.cc:401] 127.19.68.20:32857/kms/v1/keys
Aug 10 19:27:51 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390070, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
Aug 10 19:27:53 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:27:53.992513 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36019
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:36019
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:32857/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:27:54.339386 7430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:54.339675 7430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:54.339789 7430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:54.347025 7430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:27:54.347133 7430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:54.347193 7430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:54.347299 7430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:27:54.347358 7430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:27:54.357555 7430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.68.20:32857/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:36019
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:54.358779 7430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:54.360054 7430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:54.371766 7441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:54.371799 7439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:54.372020 7443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:54.372490 7430 server_base.cc:1034] running on GCE node
I20260810 19:27:54.373338 7430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:54.374774 7430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:54.375989 7430 hybrid_clock.cc:648] HybridClock initialized: now 1786390074375947 us; error 35 us; skew 500 ppm
I20260810 19:27:54.376351 7430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:54 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390074, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:54.383061 7430 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:27:54.384454 7430 webserver.cc:460] Webserver started at http://127.19.68.62:42741/ using document root <none> and password file <none>
I20260810 19:27:54.385030 7430 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:54.385164 7430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:54.385449 7430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:27:54 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390074, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:54.436832 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "3ca5d4effee44eacbe4ad7de01442506"
format_stamp: "Formatted at 2026-08-10 19:27:54 on dist-test-slave-2j7r"
server_key: "982523ae2d020cf6f6719b7a7c2e76b2"
server_key_iv: "9e7df638df836964be41d609839d23c8"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:54.437732 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "3ca5d4effee44eacbe4ad7de01442506"
format_stamp: "Formatted at 2026-08-10 19:27:54 on dist-test-slave-2j7r"
server_key: "982523ae2d020cf6f6719b7a7c2e76b2"
server_key_iv: "9e7df638df836964be41d609839d23c8"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:54.443076 7430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:27:54.462634 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:54.464282 7430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:27:54.464464 7430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "3ca5d4effee44eacbe4ad7de01442506"
format_stamp: "Formatted at 2026-08-10 19:27:54 on dist-test-slave-2j7r"
server_key: "982523ae2d020cf6f6719b7a7c2e76b2"
server_key_iv: "9e7df638df836964be41d609839d23c8"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:54.464680 7430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:54.481941 7430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:54.486001 7430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:54.486286 7430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:54.505259 7547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:36019 every 8 connection(s)
I20260810 19:27:54.505363 7430 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:36019
I20260810 19:27:54.507332 7430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:27:54.511359 7551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:54.515331 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 7430
I20260810 19:27:54.515555 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:27:54.516372 19728 external_mini_cluster.cc:1428] Setting key b20f0984072826dcdc5bb15056045c98
I20260810 19:27:54.524376 7551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: Bootstrap starting.
I20260810 19:27:54.528682 7551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:54.530151 7551 log.cc:824] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:54.533000 7551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: No bootstrap required, opened a new log
I20260810 19:27:54.541750 7551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } }
I20260810 19:27:54.542196 7551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:54.542312 7551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca5d4effee44eacbe4ad7de01442506, State: Initialized, Role: FOLLOWER
I20260810 19:27:54.542971 7551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } }
I20260810 19:27:54.543251 7551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:27:54.543437 7551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:27:54.543650 7551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 19:27:54 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:54.546428 7551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } }
I20260810 19:27:54.546972 7551 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca5d4effee44eacbe4ad7de01442506; no voters:
I20260810 19:27:54.547919 7551 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:27:54.548034 7557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:54.554798 7557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [term 1 LEADER]: Becoming Leader. State: Replica: 3ca5d4effee44eacbe4ad7de01442506, State: Running, Role: LEADER
I20260810 19:27:54.555413 7557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } }
I20260810 19:27:54.556035 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:54.518543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.19.68.62:36019)
0810 19:27:54.530058 (+ 11515us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:54.530074 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:54.530124 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:27:54.532700 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:54.532729 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:54.533250 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:54.533679 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:54.535309 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:54.537496 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:54.539309 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:54.540552 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:54.540723 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:54.545427 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:54.545438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:54.545456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:54.545507 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:54.549684 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:54.550664 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:54.550674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:54.550684 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:54.550916 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:54.551458 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:54.551463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:54.551466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:54.552256 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:54.553747 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:54.554072 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 19:27:54.554521 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11265,"thread_start_us":11013,"threads_started":1}
I20260810 19:27:54.556617 7551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:27:54.563903 7558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca5d4effee44eacbe4ad7de01442506. Latest consensus state: current_term: 1 leader_uuid: "3ca5d4effee44eacbe4ad7de01442506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } } }
I20260810 19:27:54.564003 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca5d4effee44eacbe4ad7de01442506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca5d4effee44eacbe4ad7de01442506" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36019 } } }
I20260810 19:27:54.564486 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:54.568863 7558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506 [sys.catalog]: This master's current role is: LEADER
I20260810 19:27:54.569733 7578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:27:54.576239 7578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:27:54.577386 7551 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:27:54.607788 7578 catalog_manager.cc:1347] Generated new cluster ID: 91058c990e2b46b4a0f52f6766b04eb1
I20260810 19:27:54.607952 7578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:27:54.652232 7578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:27:54.654251 7578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:27:54.663514 7578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: Generated new TSK 0
I20260810 19:27:54.664525 7578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:27:55.384996 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:32857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:27:55 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 19:27:55.824867 7636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:55.825261 7636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:55.825469 7636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:55.836984 7636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:55.837198 7636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:55.837432 7636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:27:55.853022 7636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.68.20:32857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:55.854913 7636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:55.856686 7636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:55.875356 7659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:55.875499 7660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:55.880347 7663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:55.880988 7636 server_base.cc:1034] running on GCE node
I20260810 19:27:55.881738 7636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:55.882947 7636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:55.884230 7636 hybrid_clock.cc:648] HybridClock initialized: now 1786390075884170 us; error 56 us; skew 500 ppm
I20260810 19:27:55.884560 7636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:55 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:55.887902 7636 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:27:55.889676 7636 webserver.cc:460] Webserver started at http://127.19.68.1:44893/ using document root <none> and password file <none>
I20260810 19:27:55.890405 7636 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:55.890516 7636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:55.890854 7636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:27:55 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:55.911331 7636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "6bfb5431e83a4ebea491a6a804cfc199"
format_stamp: "Formatted at 2026-08-10 19:27:55 on dist-test-slave-2j7r"
server_key: "d5a7162708304a1e38ca213a64108d03"
server_key_iv: "579caf090ce69032c7f71c246aa680b7"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:55.925444 7636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "6bfb5431e83a4ebea491a6a804cfc199"
format_stamp: "Formatted at 2026-08-10 19:27:55 on dist-test-slave-2j7r"
server_key: "d5a7162708304a1e38ca213a64108d03"
server_key_iv: "579caf090ce69032c7f71c246aa680b7"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:55.939929 7636 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260810 19:27:55.979373 7673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:55.981592 7636 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.006s sys 0.012s
I20260810 19:27:55.981906 7636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "6bfb5431e83a4ebea491a6a804cfc199"
format_stamp: "Formatted at 2026-08-10 19:27:55 on dist-test-slave-2j7r"
server_key: "d5a7162708304a1e38ca213a64108d03"
server_key_iv: "579caf090ce69032c7f71c246aa680b7"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:55.982329 7636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:56.003305 7636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:56.008807 7636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:56.009306 7636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:56.015727 7636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:56.017882 7636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:56.017999 7636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:56.018157 7636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:56.018244 7636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:56.078495 7636 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:39623
I20260810 19:27:56.078517 7872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:39623 every 8 connection(s)
I20260810 19:27:56.080436 7636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:27:56.084957 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 7636
I20260810 19:27:56.085217 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:27:56.086050 19728 external_mini_cluster.cc:1428] Setting key ff8d3c0d221a603412e00b104e3aa729
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:56.111356 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:56.084791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:44973 (local address 127.19.68.62:36019)
0810 19:27:56.089661 (+ 4870us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:56.089667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:56.092432 (+ 2765us) server_negotiation.cc:407] Connection header received
0810 19:27:56.092502 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:56.092508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:56.092600 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:56.092767 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:56.095044 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.096232 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.098795 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.099471 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.099570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:56.104106 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:56.104112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:56.104116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:56.104165 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:56.107531 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:56.108437 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:56.108442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:56.108444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:56.108606 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:56.109174 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:56.109179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:56.109182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:56.109547 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:56.109681 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:56.110687 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0810 19:27:56.110933 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4740,"thread_start_us":7923,"threads_started":1}
I20260810 19:27:56.114022 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:56.083985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36019 (local address 127.19.68.1:44973)
0810 19:27:56.089626 (+ 5641us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:56.089674 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:56.091996 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:56.093299 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:56.093312 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:56.093992 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:56.094831 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:56.094863 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.096462 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.096468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:56.098624 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:56.098641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.099921 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.099928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:56.100210 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:56.101030 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:56.101064 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:56.103857 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:56.107729 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:56.107737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:56.107754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:56.108300 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:56.108748 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:56.108752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:56.108755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:56.109021 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:56.109777 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:56.109785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:56.110214 (+ 429us) client_negotiation.cc:769] Sending connection context
0810 19:27:56.111999 (+ 1785us) client_negotiation.cc:240] Negotiation successful
0810 19:27:56.112570 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5353,"thread_start_us":8709,"threads_started":1}
I20260810 19:27:56.116211 7873 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36019
I20260810 19:27:56.116520 7873 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:56.117612 7873 heartbeater.cc:511] Master 127.19.68.62:36019 requested a full tablet report, sending...
I20260810 19:27:56.120087 7477 ts_manager.cc:194] Registered new tserver with Master: 6bfb5431e83a4ebea491a6a804cfc199 (127.19.68.1:39623)
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:56.121670 7477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:44973
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
W20260810 19:27:56.189790 19728 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.19.68.2: real 0.101s user 0.008s sys 0.017s
I20260810 19:27:56.190526 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:32857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390075, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
W20260810 19:27:56.644243 7884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:56.644773 7884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:56.644989 7884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:56.656703 7884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:56.656898 7884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:56.657158 7884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:27:56.672945 7884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.68.20:32857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:56.675014 7884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:56.676851 7884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:56.713008 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:56.714520 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:56.724855 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:56.725731 7884 server_base.cc:1034] running on GCE node
I20260810 19:27:56.726650 7884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:56.736462 7884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:56.746652 7884 hybrid_clock.cc:648] HybridClock initialized: now 1786390076746510 us; error 136 us; skew 500 ppm
I20260810 19:27:56.747104 7884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390076, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:56.751719 7884 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:27:56.777524 7884 webserver.cc:460] Webserver started at http://127.19.68.2:40485/ using document root <none> and password file <none>
I20260810 19:27:56.778692 7884 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:56.778923 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:56.779358 7884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390076, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:56.824738 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "380ef809305347ce947d90b46bee54c6"
format_stamp: "Formatted at 2026-08-10 19:27:56 on dist-test-slave-2j7r"
server_key: "583c2f6da4ec4354407251acc94245d6"
server_key_iv: "9bf44e98e2d6a2459983fd5262c6ffb5"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:56.825945 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "380ef809305347ce947d90b46bee54c6"
format_stamp: "Formatted at 2026-08-10 19:27:56 on dist-test-slave-2j7r"
server_key: "583c2f6da4ec4354407251acc94245d6"
server_key_iv: "9bf44e98e2d6a2459983fd5262c6ffb5"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:56.843181 7884 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.012s sys 0.004s
I20260810 19:27:56.889598 7931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:56.891767 7884 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.006s sys 0.000s
I20260810 19:27:56.891978 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "380ef809305347ce947d90b46bee54c6"
format_stamp: "Formatted at 2026-08-10 19:27:56 on dist-test-slave-2j7r"
server_key: "583c2f6da4ec4354407251acc94245d6"
server_key_iv: "9bf44e98e2d6a2459983fd5262c6ffb5"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:56.892390 7884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:56.915693 7884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:56.921211 7884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:56.921556 7884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:56.922782 7884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:56.924609 7884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:56.924710 7884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:56.924901 7884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:56.924973 7884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:56.971479 7884 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:44701
I20260810 19:27:56.971501 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:44701 every 8 connection(s)
I20260810 19:27:56.973330 7884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:27:56.980547 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 7884
I20260810 19:27:56.980705 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:27:56.981348 19728 external_mini_cluster.cc:1428] Setting key 721605478ec6697e6a587b86e3686ffc
Aug 10 19:27:56 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390076, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
I20260810 19:27:57.020342 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:56.977399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36019 (local address 127.19.68.2:49445)
0810 19:27:56.979767 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:56.979821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:56.982301 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:56.989268 (+ 6967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:56.989296 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:56.989925 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:56.990718 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:56.990742 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.992097 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.992102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:56.994409 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:56.994426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.995687 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.995695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:56.995978 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:56.996738 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:56.996768 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:56.999783 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:57.003579 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:57.003588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:57.003605 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:57.004154 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:57.004639 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:57.004644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:57.004647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:57.004914 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:57.005633 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:57.005642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:57.017676 (+ 12034us) client_negotiation.cc:769] Sending connection context
0810 19:27:57.018111 (+ 435us) client_negotiation.cc:240] Negotiation successful
0810 19:27:57.018765 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":1828,"threads_started":1}
I20260810 19:27:57.021652 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:56.977941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49445 (local address 127.19.68.62:36019)
0810 19:27:56.988522 (+ 10581us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:56.988528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:56.988547 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:27:56.988604 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:56.988608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:56.988699 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:56.988823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:56.990923 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.991911 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.994591 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:56.995212 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:56.995330 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.000067 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:57.000075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:57.000080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:57.000125 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:57.003403 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.004336 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.004341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.004345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.004498 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.005042 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.005046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.005049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.005414 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:57.005559 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.020927 (+ 15368us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.021240 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10459,"thread_start_us":10353,"threads_started":1}
I20260810 19:27:57.023715 8118 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36019
I20260810 19:27:57.024031 8118 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:57.024757 8118 heartbeater.cc:511] Master 127.19.68.62:36019 requested a full tablet report, sending...
I20260810 19:27:57.026327 7477 ts_manager.cc:194] Registered new tserver with Master: 380ef809305347ce947d90b46bee54c6 (127.19.68.2:44701)
I20260810 19:27:57.027395 7477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:49445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:27:57.060552 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:32857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:27:57.125123 7873 heartbeater.cc:503] Master 127.19.68.62:36019 was elected leader, sending a full tablet report...
W20260810 19:27:57.420858 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:27:57.421233 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:27:57.421360 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:27:57.428699 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:27:57.428885 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:27:57.429019 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:27:57.439065 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.68.20:32857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:27:57.440356 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:27:57.441705 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:27:57.454613 8147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:57.454640 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:27:57.455307 8140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:27:57.455754 8130 server_base.cc:1034] running on GCE node
I20260810 19:27:57.456475 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:27:57.457518 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:27:57.458719 8130 hybrid_clock.cc:648] HybridClock initialized: now 1786390077458659 us; error 50 us; skew 500 ppm
I20260810 19:27:57.459025 8130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:27:57 dist-test-slave-2j7r krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390077, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:27:57.462132 8130 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:27:57.463606 8130 webserver.cc:460] Webserver started at http://127.19.68.3:38195/ using document root <none> and password file <none>
I20260810 19:27:57.464298 8130 fs_manager.cc:359] Metadata directory not provided
I20260810 19:27:57.464427 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:27:57.464725 8130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:27:57 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390077, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:27:57.481400 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "8c04c284131344c695eda90dbda9f38e"
format_stamp: "Formatted at 2026-08-10 19:27:57 on dist-test-slave-2j7r"
server_key: "c0bfe093f3f1607d9fc56ef60f51f3fe"
server_key_iv: "6a6b8f44c14152eae4b3370a15e728fa"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:57.482388 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "8c04c284131344c695eda90dbda9f38e"
format_stamp: "Formatted at 2026-08-10 19:27:57 on dist-test-slave-2j7r"
server_key: "c0bfe093f3f1607d9fc56ef60f51f3fe"
server_key_iv: "6a6b8f44c14152eae4b3370a15e728fa"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:57.488034 8130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:27:57.505174 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:57.507000 8130 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:27:57.507177 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "8c04c284131344c695eda90dbda9f38e"
format_stamp: "Formatted at 2026-08-10 19:27:57 on dist-test-slave-2j7r"
server_key: "c0bfe093f3f1607d9fc56ef60f51f3fe"
server_key_iv: "6a6b8f44c14152eae4b3370a15e728fa"
server_key_version: "kuduclusterkey@0"
I20260810 19:27:57.507416 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:27:57.523398 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:27:57.527221 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:27:57.527472 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:27:57.528370 8130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:27:57.529703 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:27:57.529778 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:57.529874 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:27:57.529922 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:27:57.560216 8130 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:46463
I20260810 19:27:57.560278 8366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:46463 every 8 connection(s)
I20260810 19:27:57.561518 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:27:57.566237 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 8130
I20260810 19:27:57.566390 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:27:57.566809 19728 external_mini_cluster.cc:1428] Setting key ea95cab9d9db4a57b5ef44dc257bd9d4
Aug 10 19:27:57 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390077, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:27:57.580960 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.564071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:59971 (local address 127.19.68.62:36019)
0810 19:27:57.564598 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:57.564602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:57.565441 (+ 839us) server_negotiation.cc:407] Connection header received
0810 19:27:57.567290 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:57.567295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:57.567373 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:57.567474 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:57.569464 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.570315 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.571968 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.572409 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.572505 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.575720 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:57.575725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:57.575728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:57.575760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:57.578349 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.578993 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.578996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.578998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.579116 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.579445 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.579448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.579450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.579700 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:57.579805 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.580422 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.580625 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":261,"threads_started":1}
I20260810 19:27:57.582129 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.564510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36019 (local address 127.19.68.3:59971)
0810 19:27:57.565280 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:57.565329 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:57.567019 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:57.568011 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:57.568018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:57.568445 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:27:57.569265 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.569290 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.570477 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.570481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.571837 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.571850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.572630 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.572635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.572858 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.573502 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:27:57.573524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:27:57.575507 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:27:57.578463 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:57.578469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:57.578481 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:57.578903 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:57.579190 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:27:57.579193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:27:57.579195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:27:57.579365 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:27:57.579799 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:27:57.579804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:27:57.580084 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 19:27:57.580357 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 19:27:57.580796 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":312,"threads_started":1}
I20260810 19:27:57.583266 8368 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36019
I20260810 19:27:57.583541 8368 heartbeater.cc:464] Registering TS with master...
I20260810 19:27:57.584221 8368 heartbeater.cc:511] Master 127.19.68.62:36019 requested a full tablet report, sending...
I20260810 19:27:57.585568 7477 ts_manager.cc:194] Registered new tserver with Master: 8c04c284131344c695eda90dbda9f38e (127.19.68.3:46463)
I20260810 19:27:57.586360 7477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:59971
I20260810 19:27:57.594619 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:27:57.611542 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.599206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.19.68.62:36019)
0810 19:27:57.599390 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:57.599395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:57.599846 (+ 451us) server_negotiation.cc:407] Connection header received
0810 19:27:57.599969 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:57.599973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:57.600052 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:57.600139 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:57.601559 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.602379 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.603918 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.604400 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.604499 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.606878 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:57.606883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:57.606886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:57.606917 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:57.609307 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.609899 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.609903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.609905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.610020 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:57.610316 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:57.610319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:57.610321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:57.610569 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:57.610746 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.611052 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.611276 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:27:57.615321 7477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:27:57.622356 7477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:27:57.646740 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.636796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.19.68.2:44701)
0810 19:27:57.637447 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:57.637457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:57.638029 (+ 572us) server_negotiation.cc:407] Connection header received
0810 19:27:57.638232 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:57.638241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:57.639390 (+ 1149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:57.640417 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:57.640627 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.642064 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.644049 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.645491 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.645647 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.645880 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.646272 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.646437 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":395,"threads_started":1}
I20260810 19:27:57.646800 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.635671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:46463 (local address 127.0.0.1:45542)
0810 19:27:57.636587 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:57.636636 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:57.636803 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:57.639410 (+ 2607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:57.639415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:57.639445 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:57.640024 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.640035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.641782 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.641787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.644229 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.644238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.645905 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.645911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.646060 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.646092 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:27:57.646144 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 19:27:57.646334 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":150,"thread_start_us":1961,"threads_started":1}
I20260810 19:27:57.647225 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.635171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:56982)
0810 19:27:57.638261 (+ 3090us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:57.638279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:57.638403 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:57.638993 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:57.639000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:57.639036 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:57.639466 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.639481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.641488 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.641493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.644201 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.644221 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.646176 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.646184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.646375 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.646399 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:27:57.646499 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:27:57.646634 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":90,"thread_start_us":800,"threads_started":1}
I20260810 19:27:57.647568 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.635252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.19.68.1:39623)
0810 19:27:57.635835 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:57.635844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:57.638377 (+ 2533us) server_negotiation.cc:407] Connection header received
0810 19:27:57.638564 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:57.638571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:57.638781 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:57.638975 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:57.639701 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.641297 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.644406 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.645898 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.646057 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.646314 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.646555 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.646716 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260810 19:27:57.647636 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.635882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44701 (local address 127.0.0.1:60754)
0810 19:27:57.637914 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:57.637936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:57.638076 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:57.639621 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:57.639626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:57.639656 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:57.640218 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.640231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.642227 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.642233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.643879 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:57.643893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.645706 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.645713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:57.645917 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.645985 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 19:27:57.646117 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 19:27:57.647218 (+ 1101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":937,"thread_start_us":1112,"threads_started":1}
I20260810 19:27:57.648092 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:57.636609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.19.68.3:46463)
0810 19:27:57.638951 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:57.638957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:57.638985 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:27:57.639060 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:57.639065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:57.639219 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:57.639394 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:57.640228 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.641595 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.644402 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:57.645683 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:57.647327 (+ 1644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:57.647547 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:57.647613 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 19:27:57.647765 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2197,"thread_start_us":2037,"threads_started":1}
I20260810 19:27:57.651243 8012 tablet_service.cc:1467] Processing CreateTablet for tablet 09a48b4b19b444d98d53a69118315b23 (DEFAULT_TABLE table=test-table [id=0837171a9be940a496ee45c52e8aea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:57.651559 7772 tablet_service.cc:1467] Processing CreateTablet for tablet 09a48b4b19b444d98d53a69118315b23 (DEFAULT_TABLE table=test-table [id=0837171a9be940a496ee45c52e8aea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:57.651862 8251 tablet_service.cc:1467] Processing CreateTablet for tablet 09a48b4b19b444d98d53a69118315b23 (DEFAULT_TABLE table=test-table [id=0837171a9be940a496ee45c52e8aea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:27:57.652496 8012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a48b4b19b444d98d53a69118315b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:57.653116 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a48b4b19b444d98d53a69118315b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:57.653280 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a48b4b19b444d98d53a69118315b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:27:57.663990 8401 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Bootstrap starting.
I20260810 19:27:57.667227 8401 tablet_bootstrap.cc:654] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:57.668609 8401 log.cc:824] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:57.671597 8401 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: No bootstrap required, opened a new log
I20260810 19:27:57.671850 8401 ts_tablet_manager.cc:1397] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 19:27:57.673852 8402 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Bootstrap starting.
I20260810 19:27:57.670011 8403 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Bootstrap starting.
I20260810 19:27:57.677299 8402 tablet_bootstrap.cc:654] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:57.678409 8402 log.cc:824] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:57.679486 8403 tablet_bootstrap.cc:654] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Neither blocks nor log segments found. Creating new log.
I20260810 19:27:57.680686 8403 log.cc:824] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Log is configured to *not* fsync() on all Append() calls
I20260810 19:27:57.680620 8401 raft_consensus.cc:348] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.681075 8401 raft_consensus.cc:374] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:57.681193 8402 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: No bootstrap required, opened a new log
I20260810 19:27:57.681217 8401 raft_consensus.cc:729] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380ef809305347ce947d90b46bee54c6, State: Initialized, Role: FOLLOWER
I20260810 19:27:57.681558 8402 ts_tablet_manager.cc:1397] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260810 19:27:57.681905 8401 consensus_queue.cc:260] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.682955 8118 heartbeater.cc:503] Master 127.19.68.62:36019 was elected leader, sending a full tablet report...
I20260810 19:27:57.683607 8401 ts_tablet_manager.cc:1428] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260810 19:27:57.684033 8403 tablet_bootstrap.cc:492] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: No bootstrap required, opened a new log
I20260810 19:27:57.684324 8403 ts_tablet_manager.cc:1397] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20260810 19:27:57.692859 8402 raft_consensus.cc:348] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.693357 8402 raft_consensus.cc:374] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:57.693468 8402 raft_consensus.cc:729] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c04c284131344c695eda90dbda9f38e, State: Initialized, Role: FOLLOWER
I20260810 19:27:57.694114 8402 consensus_queue.cc:260] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.695231 8368 heartbeater.cc:503] Master 127.19.68.62:36019 was elected leader, sending a full tablet report...
I20260810 19:27:57.695232 8403 raft_consensus.cc:348] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.695679 8403 raft_consensus.cc:374] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:27:57.695835 8403 raft_consensus.cc:729] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bfb5431e83a4ebea491a6a804cfc199, State: Initialized, Role: FOLLOWER
I20260810 19:27:57.696076 8402 ts_tablet_manager.cc:1428] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 19:27:57.696496 8403 consensus_queue.cc:260] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:57.698268 8403 ts_tablet_manager.cc:1428] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260810 19:27:57.735025 8119 tablet.cc:2369] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:27:57.813212 8369 tablet.cc:2369] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:27:57.839437 7877 tablet.cc:2369] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:27:58.002954 8410 raft_consensus.cc:482] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:27:58.003243 8410 raft_consensus.cc:504] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:58.005020 8410 leader_election.cc:288] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bfb5431e83a4ebea491a6a804cfc199 (127.19.68.1:39623), 8c04c284131344c695eda90dbda9f38e (127.19.68.3:46463)
I20260810 19:27:58.012104 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.005497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:53201 (local address 127.19.68.3:46463)
0810 19:27:58.005693 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.005699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.006897 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 19:27:58.007100 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.007104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.007168 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.007243 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:58.007786 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.008494 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.010375 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.011166 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.011283 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.011388 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.011726 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.011852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:27:58.012249 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.005374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:46463 (local address 127.19.68.2:53201)
0810 19:27:58.006777 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.006805 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:58.006965 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:58.007269 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:58.007273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:58.007306 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:58.007644 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:58.007654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.008626 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.008630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:58.010205 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:58.010218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.011369 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.011376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:58.011522 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.011546 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:27:58.011697 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 19:27:58.011829 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":1,"thread_start_us":539,"threads_started":1}
I20260810 19:27:58.012576 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.005477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:39881 (local address 127.19.68.1:39623)
0810 19:27:58.005705 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.005711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.007097 (+ 1386us) server_negotiation.cc:407] Connection header received
0810 19:27:58.007283 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.007289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.007355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.007442 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:27:58.008074 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.008829 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.010695 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.011524 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.011653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.011776 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.012153 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.012307 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:27:58.013177 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.005601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.19.68.2:39881)
0810 19:27:58.006986 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.007001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:27:58.007138 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:27:58.007483 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:27:58.007488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:27:58.007514 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:27:58.007904 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:58.007914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.008971 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.008981 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:58.010524 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:27:58.010538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.011762 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.011769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:27:58.011921 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.011948 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:27:58.012738 (+ 790us) client_negotiation.cc:240] Negotiation successful
0810 19:27:58.012864 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":4,"thread_start_us":1088,"threads_started":1}
I20260810 19:27:58.013485 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a48b4b19b444d98d53a69118315b23" candidate_uuid: "380ef809305347ce947d90b46bee54c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c04c284131344c695eda90dbda9f38e" is_pre_election: true
I20260810 19:27:58.014092 8283 raft_consensus.cc:2454] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 380ef809305347ce947d90b46bee54c6 in term 0.
I20260810 19:27:58.014304 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a48b4b19b444d98d53a69118315b23" candidate_uuid: "380ef809305347ce947d90b46bee54c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bfb5431e83a4ebea491a6a804cfc199" is_pre_election: true
I20260810 19:27:58.014786 7936 leader_election.cc:302] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380ef809305347ce947d90b46bee54c6, 8c04c284131344c695eda90dbda9f38e; no voters:
I20260810 19:27:58.014865 7803 raft_consensus.cc:2454] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 380ef809305347ce947d90b46bee54c6 in term 0.
I20260810 19:27:58.015208 8410 raft_consensus.cc:2781] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:27:58.015391 8410 raft_consensus.cc:482] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:27:58.015525 8410 raft_consensus.cc:3037] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:58.017940 8410 raft_consensus.cc:504] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:58.018604 8410 leader_election.cc:288] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [CANDIDATE]: Term 1 election: Requested vote from peers 6bfb5431e83a4ebea491a6a804cfc199 (127.19.68.1:39623), 8c04c284131344c695eda90dbda9f38e (127.19.68.3:46463)
I20260810 19:27:58.019001 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a48b4b19b444d98d53a69118315b23" candidate_uuid: "380ef809305347ce947d90b46bee54c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bfb5431e83a4ebea491a6a804cfc199"
I20260810 19:27:58.019150 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a48b4b19b444d98d53a69118315b23" candidate_uuid: "380ef809305347ce947d90b46bee54c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c04c284131344c695eda90dbda9f38e"
I20260810 19:27:58.019248 7803 raft_consensus.cc:3037] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:58.019377 8283 raft_consensus.cc:3037] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:27:58.022713 8283 raft_consensus.cc:2454] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 380ef809305347ce947d90b46bee54c6 in term 1.
I20260810 19:27:58.022706 7803 raft_consensus.cc:2454] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 380ef809305347ce947d90b46bee54c6 in term 1.
I20260810 19:27:58.023134 7936 leader_election.cc:302] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380ef809305347ce947d90b46bee54c6, 8c04c284131344c695eda90dbda9f38e; no voters:
I20260810 19:27:58.023455 8410 raft_consensus.cc:2781] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:27:58.023926 8410 raft_consensus.cc:686] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 380ef809305347ce947d90b46bee54c6, State: Running, Role: LEADER
I20260810 19:27:58.024444 8410 consensus_queue.cc:237] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } }
I20260810 19:27:58.028959 7475 catalog_manager.cc:5495] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 380ef809305347ce947d90b46bee54c6 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "380ef809305347ce947d90b46bee54c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "380ef809305347ce947d90b46bee54c6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260810 19:27:58.102872 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.097858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.19.68.2:44701)
0810 19:27:58.098041 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.098046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.098064 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:27:58.098151 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.098155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.098247 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.098365 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:27:58.098717 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.099473 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.100828 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.101277 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.101366 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.101845 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:27:58.102373 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:27:58.102479 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.102584 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.102651 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:27:58.110829 8283 raft_consensus.cc:1261] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 1 FOLLOWER]: Refusing update from remote peer 380ef809305347ce947d90b46bee54c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:27:58.110845 7803 raft_consensus.cc:1261] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 1 FOLLOWER]: Refusing update from remote peer 380ef809305347ce947d90b46bee54c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:27:58.111747 8422 consensus_queue.cc:1035] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bfb5431e83a4ebea491a6a804cfc199" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:27:58.112133 8410 consensus_queue.cc:1035] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c04c284131344c695eda90dbda9f38e" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:27:58.125504 8432 mvcc.cc:205] Tried to move back new op lower bound from 7317053759932518400 to 7317053759591768064. Current Snapshot: MvccSnapshot[applied={T|T < 7317053759932518400 or (T in {7317053759932518400})}]
I20260810 19:27:58.136206 7475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:27:58.136471 7475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:27:58.139796 7475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ca5d4effee44eacbe4ad7de01442506: Sending DeleteTablet for 3 replicas of tablet 09a48b4b19b444d98d53a69118315b23
I20260810 19:27:58.140966 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a48b4b19b444d98d53a69118315b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:58 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:56982
I20260810 19:27:58.141047 8251 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a48b4b19b444d98d53a69118315b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:58 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:45542
I20260810 19:27:58.141129 8012 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a48b4b19b444d98d53a69118315b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:27:58 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:60754
I20260810 19:27:58.142235 8449 tablet_replica.cc:330] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: stopping tablet replica
I20260810 19:27:58.142442 8448 tablet_replica.cc:330] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: stopping tablet replica
I20260810 19:27:58.142670 8449 raft_consensus.cc:2229] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:27:58.142735 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 7636
I20260810 19:27:58.142827 8448 raft_consensus.cc:2229] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:27:58.143206 8449 raft_consensus.cc:2258] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:58.143414 8448 raft_consensus.cc:2258] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:58.145596 8448 ts_tablet_manager.cc:1905] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:27:58.151795 8448 ts_tablet_manager.cc:1918] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:58.152001 8448 log.cc:1196] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/09a48b4b19b444d98d53a69118315b23
I20260810 19:27:58.152590 8448 ts_tablet_manager.cc:1939] T 09a48b4b19b444d98d53a69118315b23 P 380ef809305347ce947d90b46bee54c6: Deleting consensus metadata
I20260810 19:27:58.154435 7456 catalog_manager.cc:4841] TS 380ef809305347ce947d90b46bee54c6 (127.19.68.2:44701): tablet 09a48b4b19b444d98d53a69118315b23 (table test-table [id=0837171a9be940a496ee45c52e8aea76]) successfully deleted
I20260810 19:27:58.155936 8449 ts_tablet_manager.cc:1905] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:27:58 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:27:58.157240 8451 tablet_replica.cc:330] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: stopping tablet replica
I20260810 19:27:58.157681 8451 raft_consensus.cc:2229] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:27:58.158195 8451 raft_consensus.cc:2258] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:27:58.160632 8451 ts_tablet_manager.cc:1905] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:27:58.164925 8449 ts_tablet_manager.cc:1918] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:58.165155 8449 log.cc:1196] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/09a48b4b19b444d98d53a69118315b23
I20260810 19:27:58.165724 8449 ts_tablet_manager.cc:1939] T 09a48b4b19b444d98d53a69118315b23 P 6bfb5431e83a4ebea491a6a804cfc199: Deleting consensus metadata
I20260810 19:27:58.166029 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.144037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.19.68.1:39623)
0810 19:27:58.144764 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.144771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.145322 (+ 551us) server_negotiation.cc:407] Connection header received
0810 19:27:58.145514 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.145519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.145599 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.145717 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:58.147439 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.148652 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.150894 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.151598 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.151697 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.157502 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:58.157514 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:58.157527 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:58.157564 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:58.161451 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.162392 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.162401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.162405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.162580 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.163061 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.163067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.163070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.163550 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:58.163838 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.165153 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.165465 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572}
I20260810 19:27:58.166988 7864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:58.167439 8451 ts_tablet_manager.cc:1918] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:27:58.167615 8451 log.cc:1196] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/09a48b4b19b444d98d53a69118315b23
I20260810 19:27:58.168326 8451 ts_tablet_manager.cc:1939] T 09a48b4b19b444d98d53a69118315b23 P 8c04c284131344c695eda90dbda9f38e: Deleting consensus metadata
I20260810 19:27:58.170378 7455 catalog_manager.cc:4841] TS 8c04c284131344c695eda90dbda9f38e (127.19.68.3:46463): tablet 09a48b4b19b444d98d53a69118315b23 (table test-table [id=0837171a9be940a496ee45c52e8aea76]) successfully deleted
I20260810 19:27:58.257370 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 7636
W20260810 19:27:58.295269 7453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:39623 (error 108)
I20260810 19:27:58.296140 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 7884
I20260810 19:27:58.297833 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.296828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57004)
0810 19:27:58.297491 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.297603 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":424,"threads_started":1}
W20260810 19:27:58.298162 7453 catalog_manager.cc:4584] TS 6bfb5431e83a4ebea491a6a804cfc199 (127.19.68.1:39623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09a48b4b19b444d98d53a69118315b23: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Aug 10 19:27:58 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:27:58.312939 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.296940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.19.68.2:44701)
0810 19:27:58.297142 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.297148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.297457 (+ 309us) server_negotiation.cc:407] Connection header received
0810 19:27:58.297543 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.297549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.297660 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.297764 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:58.299277 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.300491 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.301797 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.302449 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.302577 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.306972 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:58.306989 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:58.307003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:58.307038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:58.309968 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.310781 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.310791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.310794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.310945 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.311389 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.311393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.311395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.311719 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:58.311972 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.312356 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.312547 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:27:58.313725 8110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:58.351639 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.351175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57014)
0810 19:27:58.351386 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.351493 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 19:27:58.352666 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.352284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57028)
0810 19:27:58.352452 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.352523 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:27:58.413247 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.412740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57036)
0810 19:27:58.412948 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.413045 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 19:27:58.414309 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.413946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57046)
0810 19:27:58.414093 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.414163 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:27:58.425933 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 7884
I20260810 19:27:58.450582 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 8130
Aug 10 19:27:58 dist-test-slave-2j7r krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:27:58.465834 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.451356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.19.68.3:46463)
0810 19:27:58.451545 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:27:58.451551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:27:58.451571 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:27:58.451777 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:27:58.451781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:27:58.451870 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:27:58.451958 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:27:58.453253 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.453985 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.455318 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:27:58.455699 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:27:58.455806 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:27:58.460298 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:27:58.460312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:27:58.460326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:27:58.460360 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:27:58.463020 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.463696 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.463704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.463706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.463838 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:27:58.464253 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:27:58.464256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:27:58.464258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:27:58.464618 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:27:58.464906 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 19:27:58.465309 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 19:27:58.465531 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:27:58.466528 8362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:58.494915 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.494430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57058)
0810 19:27:58.494652 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.494749 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 19:27:58.496145 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.495665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57066)
0810 19:27:58.495879 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.495970 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 19:27:58.555406 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 8130
I20260810 19:27:58.578176 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 7430
I20260810 19:27:58.579406 7534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:27:58.692148 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:27:58.691218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:39623 (local address 127.0.0.1:57074)
0810 19:27:58.691751 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 19:27:58.691906 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260810 19:27:58.692507 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 7430
I20260810 19:27:58.726191 19728 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:27:59.070990 19728 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:27:59.071287 19728 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:27:59.416069 19728 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:27:59.416 UTC [6668] LOG: received smart shutdown request
2026-08-10 19:27:59.420 UTC [6668] LOG: background worker "logical replication launcher" (PID 6682) exited with exit code 1
2026-08-10 19:27:59.420 UTC [6673] LOG: shutting down
2026-08-10 19:27:59.572 UTC [6668] LOG: database system is shut down
2026-08-10T19:27:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49768 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:27:59 dist-test-slave-2j7r krb5kdc[8485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:27:59 dist-test-slave-2j7r krb5kdc[8485](info): set up 2 sockets
Aug 10 19:27:59 dist-test-slave-2j7r krb5kdc[8485](info): commencing operation
krb5kdc: starting...
W20260810 19:28:00.944953 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:28:01 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:28:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:28:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangeradmin_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangeradmin_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangerlookup_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangerlookup_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
I20260810 19:28:01.327170 19728 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTNmuTH/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/postgres -l logfile start
2026-08-10 19:28:03.613 UTC [8540] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 19:28:03.613 UTC [8540] LOG: listening on IPv4 address "127.19.68.20", port 44113
2026-08-10 19:28:03.616 UTC [8540] LOG: listening on Unix socket "/tmp/.s.PGSQL.44113"
2026-08-10 19:28:03.638 UTC [8545] LOG: database system was shut down at 2026-08-10 19:28:02 UTC
2026-08-10 19:28:03.642 UTC [8540] LOG: database system is ready to accept connections
I20260810 19:28:04.765668 19728 mini_postgres.cc:96] Postgres bound to 44113
2026-08-10 19:28:04.789 UTC [8561] FATAL: database "slave" does not exist
127.19.68.20:44113 - accepting connections
I20260810 19:28:04.790288 19728 mini_ranger.cc:161] Starting Ranger...
I20260810 19:28:04.821938 19728 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 19:28:05.242650 19728 mini_ranger.cc:87] Created ranger Postgres database
I20260810 19:28:05.242779 19728 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin
2026-08-10 19:28:05,524 [I] DB FLAVOR :POSTGRES
2026-08-10 19:28:05,525 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:28:05,525 [I] Checking connection..
2026-08-10 19:28:05,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 19:28:05,776 [I] Checking connection passed.
2026-08-10 19:28:05,777 [I] --------- Verifying version history table ---------
2026-08-10 19:28:05,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:28:06,017 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:28:06,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:28:06,267 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:28:06,268 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 19:28:06,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 19:28:06.522 UTC [8676] WARNING: there is no transaction in progress
2026-08-10 19:28:06,531 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 19:28:06,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:28:06,785 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 19:28:06,785 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:28:06,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 19:28:07,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 19:28:07,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2j7r.c.gcp-upstream.internal','N') ;"
2026-08-10 19:28:07,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:28:07,789 [I] Table x_portal_user does not exist in database ranger
2026-08-10 19:28:07,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:28:08,036 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 19:28:08,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:28:08,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:28:08,525 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 19:28:08,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 19:28:09.432 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.447 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.463 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.478 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.495 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.512 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.567 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.573 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.587 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.597 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.607 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.620 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.630 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.635 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.641 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:09.647 UTC [8879] WARNING: there is no transaction in progress
2026-08-10 19:28:10,191 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 19:28:10,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 19:28:10,424 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 19:28:10,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:28:10,684 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 19:28:10,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:28:10,936 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 19:28:10,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:28:11,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:28:11,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2j7r.c.gcp-upstream.internal';"
2026-08-10 19:28:11,712 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 19:28:11,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 19:28:11,975 [I] DB_PATCHES have already been applied
I20260810 19:28:11.981613 19728 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTNmuTH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 19:28:11.981721 19728 mini_ranger.cc:193] Using host: 127.19.68.20
I20260810 19:28:12.002794 19728 mini_ranger.cc:227] Ranger admin URL: http://127.19.68.20:51809
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:28:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 19:28:13.322551 19728 mini_ranger.cc:160] Time spent starting Ranger: real 8.532s user 0.002s sys 0.060s
Aug 10, 2026 7:28:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:28:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:28:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:28:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:28:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:28:34.902081 19728 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangerkms_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/rangerkms_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.68.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.68.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.68.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/HTTP_127.19.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:28:35 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390115, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:28:35.094667 19728 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:28:35.120149 19728 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:28:35.865114 19728 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:28:35.865234 19728 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms
I20260810 19:28:35.865272 19728 mini_ranger_kms.cc:227] Using postgres at 127.19.68.20:44113
2026-08-10 19:28:36,229 [I] DB FLAVOR :POSTGRES
2026-08-10 19:28:36,229 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:28:36,229 [I] Checking connection
2026-08-10 19:28:36,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:28:36,472 [I] connection success
2026-08-10 19:28:36,473 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:28:36,473 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:28:36,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:28:36,703 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:28:36,703 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:28:36,703 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:28:36,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.68.20:44113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:28:36,989 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:28:37.467013 19728 mini_ranger_kms.cc:310] Created kms service
I20260810 19:28:37.690922 19728 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:28:37.736168 19728 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:28:37 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390115, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:38.151608 19728 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:28:38.151930 19728 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTNmuTH/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTNmuTH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:28:38.151978 19728 mini_ranger_kms.cc:240] Using host: 127.19.68.20
I20260810 19:28:38.173918 19728 mini_ranger_kms.cc:276] Ranger KMS PID: 9198
I20260810 19:28:38.174036 19728 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.68.20:42411
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:28:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:28:40 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786390120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 19:28:40 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:40.846976 19728 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.752s user 0.077s sys 0.324s
I20260810 19:28:40.847242 19728 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:28:40.847412 19728 mini_ranger_kms.cc:401] 127.19.68.20:42411/kms/v1/keys
Aug 10 19:28:41 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.68.20@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
Aug 10 19:28:43 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:28:43.444319 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39351
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39351
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:28:43.784035 9291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:43.784302 9291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:43.784418 9291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:43.791803 9291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:28:43.791905 9291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:43.791960 9291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:43.792035 9291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:28:43.792096 9291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:28:43.802589 9291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.68.20:42411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39351
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:43.803901 9291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:43.805315 9291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:43.817777 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:43.817795 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:43.818387 9308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:43.819018 9291 server_base.cc:1034] running on GCE node
I20260810 19:28:43.819796 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:43.821285 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:43.822508 9291 hybrid_clock.cc:648] HybridClock initialized: now 1786390123822449 us; error 52 us; skew 500 ppm
I20260810 19:28:43.822865 9291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:43 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:43.830790 9291 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:28:43.832489 9291 webserver.cc:460] Webserver started at http://127.19.68.62:35723/ using document root <none> and password file <none>
I20260810 19:28:43.833182 9291 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:43.833309 9291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:43.833657 9291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:28:43 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:43.897185 9291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "2b913b7efaab4d4ca7420f8909e5ddb6"
format_stamp: "Formatted at 2026-08-10 19:28:43 on dist-test-slave-2j7r"
server_key: "c499db37b3efc5e6e8fc1c5b02027d54"
server_key_iv: "be024d78b47e97f88d031f0433d84918"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:43.898286 9291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "2b913b7efaab4d4ca7420f8909e5ddb6"
format_stamp: "Formatted at 2026-08-10 19:28:43 on dist-test-slave-2j7r"
server_key: "c499db37b3efc5e6e8fc1c5b02027d54"
server_key_iv: "be024d78b47e97f88d031f0433d84918"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:43.904944 9291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:28:43.926710 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:43.928174 9291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:28:43.928337 9291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2b913b7efaab4d4ca7420f8909e5ddb6"
format_stamp: "Formatted at 2026-08-10 19:28:43 on dist-test-slave-2j7r"
server_key: "c499db37b3efc5e6e8fc1c5b02027d54"
server_key_iv: "be024d78b47e97f88d031f0433d84918"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:43.928537 9291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:43.948215 9291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:43.952178 9291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:43.952458 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:43.971746 9291 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39351
I20260810 19:28:43.971781 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39351 every 8 connection(s)
I20260810 19:28:43.973054 9291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:28:43.974711 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 9291
I20260810 19:28:43.974933 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:28:43.975805 19728 external_mini_cluster.cc:1428] Setting key eeb3f11d99c5efccc2d636712828577e
I20260810 19:28:43.977643 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:43.991182 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Bootstrap starting.
Aug 10 19:28:43 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:43.994652 9413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:43.995843 9413 log.cc:824] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:43.998719 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: No bootstrap required, opened a new log
I20260810 19:28:44.004639 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:43.978380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.19.68.62:39351)
0810 19:28:43.979267 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:43.979283 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:43.979330 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:28:43.981058 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:43.981109 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:43.981605 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:43.982021 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:43.983883 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:43.985902 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:43.988020 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:43.989108 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:43.989323 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:43.994440 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:43.994453 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:43.994473 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:43.994525 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:43.999055 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:43.999963 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:43.999973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:43.999982 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:44.000205 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:44.000658 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:44.000664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:44.000668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:44.001584 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:44.001970 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:44.002345 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 19:28:44.003027 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":494,"threads_started":1}
I20260810 19:28:44.007431 9413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:44.007903 9413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:44.008102 9413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b913b7efaab4d4ca7420f8909e5ddb6, State: Initialized, Role: FOLLOWER
I20260810 19:28:44.008631 9413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:44.008847 9413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:28:44.008977 9413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:28:44.009147 9413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:44.011260 9413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:44.011869 9413 leader_election.cc:302] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b913b7efaab4d4ca7420f8909e5ddb6; no voters:
I20260810 19:28:44.012538 9413 leader_election.cc:288] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:28:44.012648 9421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:28:44.014914 9413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:28:44.017417 9421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b913b7efaab4d4ca7420f8909e5ddb6, State: Running, Role: LEADER
I20260810 19:28:44.018079 9421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:44.021039 9426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } } }
I20260810 19:28:44.021857 9426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:44.021436 9423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b913b7efaab4d4ca7420f8909e5ddb6. Latest consensus state: current_term: 1 leader_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } } }
I20260810 19:28:44.022541 9423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:44.026216 9437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:28:44.033170 9437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:28:44.051203 9413 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:28:44.051664 9437 catalog_manager.cc:1347] Generated new cluster ID: 06a5f565599a410a911f919f313229c3
I20260810 19:28:44.051800 9437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:28:44.075241 9437 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:28:44.078883 9437 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:28:44.098279 9437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Generated new TSK 0
I20260810 19:28:44.099490 9437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:28:44.963014 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:28:45 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 19:28:45.360759 9488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:45.361166 9488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:45.361333 9488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:45.372673 9488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:45.372807 9488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:45.372992 9488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:28:45.384294 9488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:45.386036 9488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:45.387845 9488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:45.413753 9500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:45.418849 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:45.422509 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:45.423239 9488 server_base.cc:1034] running on GCE node
I20260810 19:28:45.424094 9488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:45.432408 9488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:45.433732 9488 hybrid_clock.cc:648] HybridClock initialized: now 1786390125433646 us; error 75 us; skew 500 ppm
I20260810 19:28:45.434136 9488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:45 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:45.438656 9488 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:28:45.441740 9488 webserver.cc:460] Webserver started at http://127.19.68.1:40991/ using document root <none> and password file <none>
I20260810 19:28:45.442757 9488 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:45.442914 9488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:45.443320 9488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:28:45 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:45.466827 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "40cff868c18a45cda5c8a4b8070d241b"
format_stamp: "Formatted at 2026-08-10 19:28:45 on dist-test-slave-2j7r"
server_key: "62b7d9d5b4296ac9a6e5266cf2644e97"
server_key_iv: "f25a6d621b2db98ba84b05e632b40911"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:45.468138 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "40cff868c18a45cda5c8a4b8070d241b"
format_stamp: "Formatted at 2026-08-10 19:28:45 on dist-test-slave-2j7r"
server_key: "62b7d9d5b4296ac9a6e5266cf2644e97"
server_key_iv: "f25a6d621b2db98ba84b05e632b40911"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:45.475625 9488 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260810 19:28:45.492172 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:45.493736 9488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:28:45.493937 9488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "40cff868c18a45cda5c8a4b8070d241b"
format_stamp: "Formatted at 2026-08-10 19:28:45 on dist-test-slave-2j7r"
server_key: "62b7d9d5b4296ac9a6e5266cf2644e97"
server_key_iv: "f25a6d621b2db98ba84b05e632b40911"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:45.494155 9488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:45.541019 9488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:45.546463 9488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:45.546885 9488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:45.554337 9488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:45.562573 9488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:45.562741 9488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 19:28:45.562952 9488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:45.563076 9488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:28:45 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:46.221120 9715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:46651 every 8 connection(s)
I20260810 19:28:46.221508 9488 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:46651
I20260810 19:28:46.223363 9488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:28:46.231114 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 9488
I20260810 19:28:46.231349 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:28:46.232265 19728 external_mini_cluster.cc:1428] Setting key 489df3ff9e0340e38ccf0c46d84e64bd
Aug 10 19:28:46 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
Aug 10 19:28:46 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:46.272964 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:46.231515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.1:39949)
0810 19:28:46.244937 (+ 13422us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:46.244997 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:46.247699 (+ 2702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:46.253860 (+ 6161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:46.253875 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:46.254547 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:46.255375 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:46.255402 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.256980 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.256986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:46.258984 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:46.259011 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.260215 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.260235 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:46.260502 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:46.261286 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:46.261322 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:46.264125 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:46.268193 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:46.268203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:46.268234 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:46.268898 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:46.269421 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:46.269426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:46.269430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:46.269680 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:46.270355 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:46.270363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:46.270803 (+ 440us) client_negotiation.cc:769] Sending connection context
0810 19:28:46.271090 (+ 287us) client_negotiation.cc:240] Negotiation successful
0810 19:28:46.271615 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13054,"thread_start_us":17174,"threads_started":1}
I20260810 19:28:46.274968 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:46.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:39949 (local address 127.19.68.62:39351)
0810 19:28:46.253258 (+ 20160us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:46.253264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:46.253285 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:28:46.253345 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:46.253349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:46.253422 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:46.253530 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:46.255608 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.256773 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.259171 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.259808 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.259921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:46.264383 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:46.264390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:46.264395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:46.264447 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:46.268024 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:46.269075 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:46.269102 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:46.269105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:46.269282 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:46.269833 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:46.269838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:46.269841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:46.270188 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:46.274201 (+ 4013us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:46.274352 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 19:28:46.274576 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17362,"thread_start_us":15882,"threads_started":1}
I20260810 19:28:46.276577 9716 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:46.276887 9716 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:46.277769 9716 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:46.280297 9348 ts_manager.cc:194] Registered new tserver with Master: 40cff868c18a45cda5c8a4b8070d241b (127.19.68.1:46651)
I20260810 19:28:46.282364 9348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:39949
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
W20260810 19:28:46.359671 19728 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.19.68.2: real 0.109s user 0.003s sys 0.022s
I20260810 19:28:46.360569 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:28:46.777235 9729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:46.777510 9729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:46.777627 9729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:46.784811 9729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:46.784916 9729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:46.785060 9729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:28:46.794730 9729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:46.796020 9729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:46.797371 9729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:46.815310 9745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:46.819339 9746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:46.821283 9729 server_base.cc:1034] running on GCE node
W20260810 19:28:46.821596 9750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:46.822394 9729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:46.823698 9729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:46.824926 9729 hybrid_clock.cc:648] HybridClock initialized: now 1786390126824857 us; error 48 us; skew 500 ppm
I20260810 19:28:46.825394 9729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:46 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:46.829501 9729 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:28:46.831377 9729 webserver.cc:460] Webserver started at http://127.19.68.2:46049/ using document root <none> and password file <none>
I20260810 19:28:46.832187 9729 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:46.832348 9729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:46.832644 9729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:28:46 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:46.850834 9729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "1b12902f853f40608c87221fec246b7c"
format_stamp: "Formatted at 2026-08-10 19:28:46 on dist-test-slave-2j7r"
server_key: "4e05250adeadc242147e6d01acecb2c6"
server_key_iv: "4ee6c78a364a153e28fe7768f4c45e34"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:46.852072 9729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "1b12902f853f40608c87221fec246b7c"
format_stamp: "Formatted at 2026-08-10 19:28:46 on dist-test-slave-2j7r"
server_key: "4e05250adeadc242147e6d01acecb2c6"
server_key_iv: "4ee6c78a364a153e28fe7768f4c45e34"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:46.859535 9729 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 19:28:46.880353 9757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:46.882364 9729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:28:46.882591 9729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "1b12902f853f40608c87221fec246b7c"
format_stamp: "Formatted at 2026-08-10 19:28:46 on dist-test-slave-2j7r"
server_key: "4e05250adeadc242147e6d01acecb2c6"
server_key_iv: "4ee6c78a364a153e28fe7768f4c45e34"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:46.882820 9729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:46.912528 9729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:46.916653 9729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:46.916949 9729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:46.918012 9729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:46.919406 9729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:46.919486 9729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:46.919601 9729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:46.919658 9729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:46.955760 9729 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:43853
I20260810 19:28:46.955797 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:43853 every 8 connection(s)
I20260810 19:28:46.957003 9729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:28:46.958770 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 9729
I20260810 19:28:46.958953 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:28:46.959512 19728 external_mini_cluster.cc:1428] Setting key 642f0f20f487e8683e54472b86c698ec
Aug 10 19:28:46 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:46.977576 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:46.959519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:54611 (local address 127.19.68.62:39351)
0810 19:28:46.959990 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:46.959995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:46.961250 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 19:28:46.963031 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:46.963038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:46.963112 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:46.963228 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:46.964549 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.965535 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.967112 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.967676 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.967796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:46.971026 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:46.971033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:46.971037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:46.971076 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:46.974242 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:46.975084 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:46.975088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:46.975091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:46.975217 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:46.975729 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:46.975734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:46.975737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:46.976040 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:46.976149 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:46.976959 (+ 810us) server_negotiation.cc:299] Negotiation successful
0810 19:28:46.977183 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260810 19:28:46.978726 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:46.960170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.2:54611)
0810 19:28:46.961055 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:46.961133 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:46.962764 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:46.963401 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:46.963408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:46.963807 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:46.964376 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:46.964394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.965686 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.965690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:46.966957 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:46.966977 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:46.967851 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:46.967855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:46.968067 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:46.968689 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:46.968710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:46.970808 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:46.974373 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:46.974382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:46.974397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:46.974955 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:46.975359 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:46.975364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:46.975367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:46.975621 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:46.976152 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:46.976160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:46.976533 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 19:28:46.976828 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 19:28:46.977358 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
I20260810 19:28:46.980120 9939 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:46.980434 9939 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:46.981221 9939 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:46.982717 9348 ts_manager.cc:194] Registered new tserver with Master: 1b12902f853f40608c87221fec246b7c (127.19.68.2:43853)
I20260810 19:28:46.983495 9348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:54611
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:28:47.026443 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:28:47.286250 9716 heartbeater.cc:503] Master 127.19.68.62:39351 was elected leader, sending a full tablet report...
W20260810 19:28:47.362257 9946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:47.362534 9946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:47.362660 9946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:47.369755 9946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:47.369864 9946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:47.370021 9946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:28:47.380074 9946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:47.381372 9946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:47.382619 9946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:47.395430 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:47.395426 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:47.395493 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:47.396441 9946 server_base.cc:1034] running on GCE node
I20260810 19:28:47.397070 9946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:47.398129 9946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:47.399356 9946 hybrid_clock.cc:648] HybridClock initialized: now 1786390127399274 us; error 77 us; skew 500 ppm
I20260810 19:28:47.399669 9946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:47.402900 9946 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:28:47.404351 9946 webserver.cc:460] Webserver started at http://127.19.68.3:43937/ using document root <none> and password file <none>
I20260810 19:28:47.404999 9946 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:47.405150 9946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:47.405438 9946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:47.422055 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "e6a56328cbc644afb3552a5c3315b064"
format_stamp: "Formatted at 2026-08-10 19:28:47 on dist-test-slave-2j7r"
server_key: "d865830608199a73839b8ac9f4f94901"
server_key_iv: "27aa61c2c49663ef4f3066f702eab3a9"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:47.422899 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "e6a56328cbc644afb3552a5c3315b064"
format_stamp: "Formatted at 2026-08-10 19:28:47 on dist-test-slave-2j7r"
server_key: "d865830608199a73839b8ac9f4f94901"
server_key_iv: "27aa61c2c49663ef4f3066f702eab3a9"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:47.428527 9946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:28:47.444792 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:47.446550 9946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:28:47.446738 9946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "e6a56328cbc644afb3552a5c3315b064"
format_stamp: "Formatted at 2026-08-10 19:28:47 on dist-test-slave-2j7r"
server_key: "d865830608199a73839b8ac9f4f94901"
server_key_iv: "27aa61c2c49663ef4f3066f702eab3a9"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:47.446998 9946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:47.473505 9946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:47.478412 9946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:47.478758 9946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:47.479974 9946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:47.481835 9946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:47.481941 9946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:47.482086 9946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:47.482159 9946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:47.517534 9946 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:40493
I20260810 19:28:47.517551 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:40493 every 8 connection(s)
I20260810 19:28:47.519137 9946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:28:47.522308 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 9946
I20260810 19:28:47.522492 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:28:47.523097 19728 external_mini_cluster.cc:1428] Setting key f24fa92c2233b059a9b1a0e3ded3632b
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:47.545140 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:47.521823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:34017 (local address 127.19.68.62:39351)
0810 19:28:47.529481 (+ 7658us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:47.529487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:47.529507 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:28:47.530107 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:47.530113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:47.530206 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:47.530331 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:47.531696 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.532896 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.534518 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.535208 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.535360 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:47.538900 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:47.538907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:47.538912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:47.538957 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:47.542236 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.542909 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.542915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.542919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.543067 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.543444 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.543448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.543452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.543787 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:47.543932 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:47.544518 (+ 586us) server_negotiation.cc:299] Negotiation successful
0810 19:28:47.544744 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7535,"thread_start_us":7389,"threads_started":1}
I20260810 19:28:47.545899 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:47.522424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.3:34017)
0810 19:28:47.527566 (+ 5142us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:47.527633 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:47.529872 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:47.530527 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:47.530534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:47.530939 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:47.531537 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:47.531556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.533062 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.533067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:47.534358 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:47.534380 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.535418 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.535423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:47.535616 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:47.536246 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:47.536267 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:47.538667 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:47.542370 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:47.542377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:47.542389 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:47.542798 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:47.543154 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:47.543157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:47.543159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:47.543345 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:47.543888 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:47.543909 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:47.544193 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 19:28:47.544403 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 19:28:47.544763 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4813,"thread_start_us":4593,"threads_started":1}
I20260810 19:28:47.547271 10157 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:47.547541 10157 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:47.548203 10157 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:47.549642 9348 ts_manager.cc:194] Registered new tserver with Master: e6a56328cbc644afb3552a5c3315b064 (127.19.68.3:40493)
I20260810 19:28:47.550647 9348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:34017
I20260810 19:28:47.551775 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:28:47.553974 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 9488
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:28:47.569900 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:47.554883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.19.68.1:46651)
0810 19:28:47.555572 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:47.555578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:47.555604 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:28:47.555662 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:47.555668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:47.555857 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:47.555984 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:47.557528 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.558615 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.559961 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.560555 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.560666 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:47.564593 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:47.564604 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:47.564616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:47.564650 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:47.567301 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.568017 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.568025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.568027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.568172 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.568488 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.568491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.568493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.568809 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:47.569057 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:47.569388 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 19:28:47.569583 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":352,"threads_started":1}
I20260810 19:28:47.571211 9708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:47.653198 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 9488
I20260810 19:28:47.675436 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 9729
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:28:47.691720 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:47.676502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.19.68.2:43853)
0810 19:28:47.677183 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:47.677193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:47.677229 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:28:47.677304 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:47.677311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:47.677544 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:47.677682 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:47.679039 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.680060 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.681515 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.682058 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.682154 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:47.686488 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:47.686499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:47.686513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:47.686553 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:47.689338 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.689998 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.690005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.690007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.690137 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.690483 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.690486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.690488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.690809 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:47.691042 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:47.691231 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 19:28:47.691433 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":297,"threads_started":1}
I20260810 19:28:47.693126 9929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:47.773940 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 9729
I20260810 19:28:47.792829 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 9946
Aug 10 19:28:47 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:28:47.809020 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:47.793870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.19.68.3:40493)
0810 19:28:47.794584 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:47.794591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:47.794625 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:28:47.794691 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:47.794699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:47.794923 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:47.795065 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:47.796362 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.797566 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.798897 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:47.799505 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:47.799620 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:47.803867 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:47.803879 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:47.803894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:47.803931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:47.806673 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.807321 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.807329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.807332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.807463 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:47.807729 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:47.807732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:47.807734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:47.808080 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:47.808359 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:47.808478 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 19:28:47.808670 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":369,"threads_started":1}
I20260810 19:28:47.810282 10148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:47.892084 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 9946
I20260810 19:28:47.914321 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 9291
I20260810 19:28:47.915630 9403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:48.021018 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 9291
I20260810 19:28:48.049688 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39351
--webserver_interface=127.19.68.62
--webserver_port=35723
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39351
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:28:48.377205 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:48.377476 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:48.377590 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:48.384665 10180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:28:48.384761 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:48.384824 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:48.384908 10180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:28:48.384974 10180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:28:48.394665 10180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39351
--ranger_config_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=35723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:48.395910 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:48.397303 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:48.408922 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:48.409001 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:48.408967 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:48.410152 10180 server_base.cc:1034] running on GCE node
I20260810 19:28:48.410849 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:48.412175 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:48.413369 10180 hybrid_clock.cc:648] HybridClock initialized: now 1786390128413322 us; error 36 us; skew 500 ppm
I20260810 19:28:48.413699 10180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:48 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:48.416795 10180 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:28:48.418303 10180 webserver.cc:460] Webserver started at http://127.19.68.62:35723/ using document root <none> and password file <none>
I20260810 19:28:48.419019 10180 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:48.419157 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:28:48 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:48.506283 10180 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:28:48.509558 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:48.511034 10180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:28:48.511209 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2b913b7efaab4d4ca7420f8909e5ddb6"
format_stamp: "Formatted at 2026-08-10 19:28:43 on dist-test-slave-2j7r"
server_key: "c499db37b3efc5e6e8fc1c5b02027d54"
server_key_iv: "be024d78b47e97f88d031f0433d84918"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:48.512157 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:48.538659 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:48.542526 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:48.542810 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:48.560787 10180 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39351
I20260810 19:28:48.560806 10301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39351 every 8 connection(s)
I20260810 19:28:48.562243 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:28:48.568733 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 10180
I20260810 19:28:48.569007 10302 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:28:48.569669 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:46651
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:39351
--webserver_port=40991
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:28:48.572299 10302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Bootstrap starting.
I20260810 19:28:48.590842 10302 log.cc:824] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:48.599774 10302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:28:48.600278 10302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Bootstrap complete.
I20260810 19:28:48.609481 10302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:48.610317 10302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b913b7efaab4d4ca7420f8909e5ddb6, State: Initialized, Role: FOLLOWER
I20260810 19:28:48.610944 10302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:48.611160 10302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:28:48.611274 10302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:28:48.611435 10302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:28:48.615120 10302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:48.615687 10302 leader_election.cc:302] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b913b7efaab4d4ca7420f8909e5ddb6; no voters:
I20260810 19:28:48.616258 10302 leader_election.cc:288] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:28:48.616442 10308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:28:48.618201 10302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:28:48.621356 10308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [term 2 LEADER]: Becoming Leader. State: Replica: 2b913b7efaab4d4ca7420f8909e5ddb6, State: Running, Role: LEADER
I20260810 19:28:48.621925 10308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } }
I20260810 19:28:48.624589 10309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } } }
I20260810 19:28:48.624984 10309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:48.625294 10313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b913b7efaab4d4ca7420f8909e5ddb6. Latest consensus state: current_term: 2 leader_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b913b7efaab4d4ca7420f8909e5ddb6" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39351 } } }
I20260810 19:28:48.625574 10313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:48.634414 10329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:28:48.640100 10329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:28:48.640970 10302 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:28:48.641628 10329 catalog_manager.cc:1259] Loaded cluster ID: 06a5f565599a410a911f919f313229c3
I20260810 19:28:48.641726 10329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:28:48.654394 10329 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:28:48.656662 10329 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Loaded TSK: 0
I20260810 19:28:48.658390 10329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:28:48.986932 10305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:48.987313 10305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:48.987469 10305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:48.998632 10305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:48.998771 10305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:48.998966 10305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:28:49.013847 10305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:46651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=40991
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:49.015482 10305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:49.017208 10305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:49.039827 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:49.046682 10305 server_base.cc:1034] running on GCE node
W20260810 19:28:49.047251 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:49.047573 10370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:49.048436 10305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:49.053411 10305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:49.054693 10305 hybrid_clock.cc:648] HybridClock initialized: now 1786390129054615 us; error 66 us; skew 500 ppm
I20260810 19:28:49.055132 10305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:49.062525 10305 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:28:49.065413 10305 webserver.cc:460] Webserver started at http://127.19.68.1:40991/ using document root <none> and password file <none>
I20260810 19:28:49.066442 10305 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:49.066604 10305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:49.105188 10305 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.005s sys 0.006s
I20260810 19:28:49.118510 10381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.122807 10305 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.012s
I20260810 19:28:49.123131 10305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "40cff868c18a45cda5c8a4b8070d241b"
format_stamp: "Formatted at 2026-08-10 19:28:45 on dist-test-slave-2j7r"
server_key: "62b7d9d5b4296ac9a6e5266cf2644e97"
server_key_iv: "f25a6d621b2db98ba84b05e632b40911"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:49.124773 10305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:49.184942 10305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:49.190680 10305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:49.191120 10305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:49.198809 10305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:49.205636 10305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:49.205796 10305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.205935 10305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:49.206053 10305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.264240 10305 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:46651
I20260810 19:28:49.264297 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:46651 every 8 connection(s)
I20260810 19:28:49.266175 10305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:28:49.271901 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 10305
I20260810 19:28:49.272764 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:43853
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:39351
--webserver_port=46049
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:49.295908 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:49.270116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.1:58883)
0810 19:28:49.271031 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:49.271076 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:49.273365 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:49.275689 (+ 2324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:49.275699 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:49.276263 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:49.277026 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:49.277051 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:49.279278 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:49.279283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:49.281139 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:49.281154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:49.282802 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:49.282809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:49.283079 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:49.283778 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:49.283807 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:49.286134 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:49.290467 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:49.290476 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:49.290493 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:49.291122 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:49.291730 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:49.291735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:49.291738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:49.291996 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:49.292999 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:49.293007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:49.293488 (+ 481us) client_negotiation.cc:769] Sending connection context
0810 19:28:49.293772 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 19:28:49.294408 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":467,"threads_started":1}
I20260810 19:28:49.295918 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:49.269976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58883 (local address 127.19.68.62:39351)
0810 19:28:49.270686 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:49.270706 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:49.271215 (+ 509us) server_negotiation.cc:407] Connection header received
0810 19:28:49.274087 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:49.274107 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:49.275115 (+ 1008us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:49.275525 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:49.277302 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:49.279061 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:49.281302 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:49.282249 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:49.282425 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:49.286466 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:49.286477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:49.286496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:49.286544 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:49.290279 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:49.291319 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:49.291328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:49.291335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:49.291559 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:49.292148 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:49.292153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:49.292156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:49.292831 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:49.293044 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:49.294145 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0810 19:28:49.294563 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260810 19:28:49.299554 10572 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:49.299882 10572 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:49.300854 10572 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
W20260810 19:28:49.304306 10572 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39351 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:28:49.304543 10572 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:49.304884 10572 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:49.306226 10572 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:49.306519 10572 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
W20260810 19:28:49.307585 10572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:28:49.687281 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:49.687681 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:49.687935 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:49.698638 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:49.698757 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:49.698935 10576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:28:49.714632 10576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:43853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=46049
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:49.716148 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:49.717648 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:49.750712 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:49.754276 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:49.756879 10576 server_base.cc:1034] running on GCE node
W20260810 19:28:49.757226 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:49.758142 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:49.759539 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:49.762955 10576 hybrid_clock.cc:648] HybridClock initialized: now 1786390129762877 us; error 58 us; skew 500 ppm
I20260810 19:28:49.763298 10576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:49.767510 10576 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:28:49.790462 10576 webserver.cc:460] Webserver started at http://127.19.68.2:46049/ using document root <none> and password file <none>
I20260810 19:28:49.791280 10576 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:49.791420 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:49.830467 10576 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.017s sys 0.000s
I20260810 19:28:49.840301 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.842378 10576 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.001s
I20260810 19:28:49.842569 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "1b12902f853f40608c87221fec246b7c"
format_stamp: "Formatted at 2026-08-10 19:28:46 on dist-test-slave-2j7r"
server_key: "4e05250adeadc242147e6d01acecb2c6"
server_key_iv: "4ee6c78a364a153e28fe7768f4c45e34"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:49.843614 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 19:28:49 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:49.898350 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:49.902527 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:49.902820 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:49.903875 10576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:49.905329 10576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:49.905424 10576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.905537 10576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:49.905606 10576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:49.948719 10576 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:43853
I20260810 19:28:49.948801 10815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:43853 every 8 connection(s)
I20260810 19:28:49.950718 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:28:49.960129 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 10576
I20260810 19:28:49.960911 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.68.20:42411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:40493
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:39351
--webserver_port=43937
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:28:50 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:50.018149 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:49.954877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:44611 (local address 127.19.68.62:39351)
0810 19:28:49.988063 (+ 33186us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:49.988071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:49.992138 (+ 4067us) server_negotiation.cc:407] Connection header received
0810 19:28:49.992223 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:49.992229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:49.992326 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:49.992439 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:50.001268 (+ 8829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.002420 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.005014 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.005712 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.005843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:50.010626 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:50.010633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:50.010638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:50.010680 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:50.014194 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:50.015151 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:50.015156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:50.015160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:50.015322 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:50.015944 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:50.015949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:50.015953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:50.016330 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:50.016464 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:50.017460 (+ 996us) server_negotiation.cc:299] Negotiation successful
0810 19:28:50.017693 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33019,"thread_start_us":33212,"threads_started":1}
I20260810 19:28:50.020758 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:49.954221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.2:44611)
0810 19:28:49.988870 (+ 34649us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:49.988937 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:49.991613 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:49.998343 (+ 6730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:49.998358 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:49.999086 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:49.999943 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:49.999969 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.002652 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.002657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:50.004823 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:50.004839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.006283 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.006293 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:50.006979 (+ 686us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:50.007724 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:50.007770 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:50.010348 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:50.014366 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:50.014374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:50.014390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:50.014999 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:50.015484 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:50.015488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:50.015492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:50.015775 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:50.016569 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:50.016577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:50.016989 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 19:28:50.018692 (+ 1703us) client_negotiation.cc:240] Negotiation successful
0810 19:28:50.019429 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34277,"thread_start_us":42761,"threads_started":1}
I20260810 19:28:50.022722 10817 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:50.023080 10817 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:50.023941 10817 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:50.026113 10235 ts_manager.cc:194] Registered new tserver with Master: 1b12902f853f40608c87221fec246b7c (127.19.68.2:43853)
I20260810 19:28:50.028308 10235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:44611
Aug 10 19:28:50 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:50.307739 10572 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:50.308169 10572 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:50.309499 10235 ts_manager.cc:194] Registered new tserver with Master: 40cff868c18a45cda5c8a4b8070d241b (127.19.68.1:46651)
I20260810 19:28:50.310662 10235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:58883
W20260810 19:28:50.455076 10823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:50.455590 10823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:50.455796 10823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:50.467092 10823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:50.467286 10823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:50.467511 10823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:28:50.482920 10823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.68.20:42411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:40493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=43937
--tserver_master_addrs=127.19.68.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:50.484956 10823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:50.486953 10823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:50.522238 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:50.528683 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:50.530315 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:50.530848 10823 server_base.cc:1034] running on GCE node
I20260810 19:28:50.532020 10823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:50.538595 10823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:50.539886 10823 hybrid_clock.cc:648] HybridClock initialized: now 1786390130539779 us; error 103 us; skew 500 ppm
I20260810 19:28:50.540349 10823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:50 dist-test-slave-2j7r krb5kdc[8485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390130, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:50.544978 10823 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:28:50.560995 10823 webserver.cc:460] Webserver started at http://127.19.68.3:43937/ using document root <none> and password file <none>
I20260810 19:28:50.562038 10823 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:50.562268 10823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:28:50 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390130, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:50.602866 10823 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.002s sys 0.011s
I20260810 19:28:50.613806 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:50.615975 10823 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.001s
I20260810 19:28:50.616227 10823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "e6a56328cbc644afb3552a5c3315b064"
format_stamp: "Formatted at 2026-08-10 19:28:47 on dist-test-slave-2j7r"
server_key: "d865830608199a73839b8ac9f4f94901"
server_key_iv: "27aa61c2c49663ef4f3066f702eab3a9"
server_key_version: "kuduclusterkey@0"
I20260810 19:28:50.617954 10823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:50.682343 10823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:50.688261 10823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:50.688719 10823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:50.693639 10823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:28:50.710140 10823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:28:50.710377 10823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 19:28:50.710567 10823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:28:50.710837 10823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:28:50 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for HTTP/127.19.68.20@KRBTEST.COM
I20260810 19:28:50.942978 10823 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:40493
I20260810 19:28:50.943043 11029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:40493 every 8 connection(s)
I20260810 19:28:50.944818 10823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:28:50.947256 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 10823
I20260810 19:28:51.003927 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:50.954889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.19.68.62:39351)
0810 19:28:50.980223 (+ 25334us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:50.980230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:50.980258 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:28:50.980336 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:50.980342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:50.980446 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:50.980754 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:50.983128 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.984418 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.986523 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:50.987158 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:50.987275 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:50.991198 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:50.991206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:50.991211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:50.991256 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:50.998987 (+ 7731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:50.999742 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:50.999747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:50.999751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:50.999909 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.002179 (+ 2270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.002184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.002188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.002661 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.002959 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.003313 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.003553 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25174,"thread_start_us":42197,"threads_started":1}
Aug 10 19:28:51 dist-test-slave-2j7r krb5kdc[8485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390130, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:51.022533 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:50.957638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39351 (local address 127.19.68.3:60183)
0810 19:28:50.995403 (+ 37765us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:50.995482 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:50.998260 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.000038 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.000052 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.000670 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:51.001624 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.001654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.005741 (+ 4087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.005747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.008306 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.008325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.009504 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.009514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.009832 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.010595 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:51.010634 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:51.013286 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:51.017501 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:51.017511 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:51.017530 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:51.018169 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:51.018617 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:51.018622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:51.018626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:51.018912 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:51.019754 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:51.019773 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:51.020330 (+ 557us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.020629 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.021309 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37418,"thread_start_us":37189,"threads_started":1}
I20260810 19:28:51.023800 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:50.957434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:60183 (local address 127.19.68.62:39351)
0810 19:28:50.994752 (+ 37318us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:50.994757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:50.997191 (+ 2434us) server_negotiation.cc:407] Connection header received
0810 19:28:50.998705 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:50.998711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:50.998825 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:50.998939 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:51.002367 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.004947 (+ 2580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.008480 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.009210 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.009351 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.013593 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:51.013601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:51.013606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:51.013644 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:51.017344 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.018334 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.018339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.018342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.018485 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.019096 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.019101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.019105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.019538 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.019679 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.023336 (+ 3657us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.023503 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37177,"thread_start_us":23166,"threads_started":1}
I20260810 19:28:51.025025 11030 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39351
I20260810 19:28:51.025429 11030 heartbeater.cc:464] Registering TS with master...
I20260810 19:28:51.026168 11030 heartbeater.cc:511] Master 127.19.68.62:39351 requested a full tablet report, sending...
I20260810 19:28:51.029618 10235 ts_manager.cc:194] Registered new tserver with Master: e6a56328cbc644afb3552a5c3315b064 (127.19.68.3:40493)
I20260810 19:28:51.030738 10235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:60183
I20260810 19:28:51.032547 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:28:51.032938 10817 heartbeater.cc:503] Master 127.19.68.62:39351 was elected leader, sending a full tablet report...
I20260810 19:28:51.050582 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.036978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.19.68.62:39351)
0810 19:28:51.037165 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.037171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.037685 (+ 514us) server_negotiation.cc:407] Connection header received
0810 19:28:51.037858 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.037863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.037943 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.038024 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:51.039649 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.040349 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.042076 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.042525 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.042645 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.045118 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:51.045125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:51.045130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:51.045168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:51.047793 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.048480 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.048486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.048489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.048621 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.049026 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.049031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.049034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.049368 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.049589 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.049930 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.050137 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:28:51.055449 10235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:28:51.062933 10235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:28:51.099680 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.077836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.19.68.3:40493)
0810 19:28:51.090753 (+ 12917us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.090761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.090795 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:28:51.090885 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.090892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.091108 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.092164 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:28:51.092464 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.094216 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.096691 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.098109 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.098431 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.098695 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.099158 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.099331 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12731,"thread_start_us":13400,"threads_started":1}
I20260810 19:28:51.099730 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.077253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:40493 (local address 127.0.0.1:55878)
0810 19:28:51.084398 (+ 7145us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.084456 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:51.084653 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.091493 (+ 6840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.091498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.091526 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:28:51.091949 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.091960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.094369 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.094375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.096529 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.096540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.098596 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.098605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.098790 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.098833 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.098946 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.099132 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":733,"threads_started":1}
I20260810 19:28:51.100359 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.077558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:43853 (local address 127.0.0.1:40218)
0810 19:28:51.088257 (+ 10699us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.088276 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:51.088390 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.090600 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.090606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.090637 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:28:51.091085 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.091099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.093832 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.093838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.096351 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.096367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.099482 (+ 3115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.099489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.099751 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.099785 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.099885 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.100011 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10554,"thread_start_us":17727,"threads_started":1}
I20260810 19:28:51.101243 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.078662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.19.68.2:43853)
0810 19:28:51.088971 (+ 10309us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.088981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.089013 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:28:51.089148 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.089158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.089446 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.089658 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:28:51.092045 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.093662 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.096550 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.098014 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.098141 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.098409 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.100686 (+ 2277us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.100868 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10103,"thread_start_us":9934,"threads_started":1}
I20260810 19:28:51.104437 10928 tablet_service.cc:1467] Processing CreateTablet for tablet aaba62d6395045f78fbc04a87d5a06ea (DEFAULT_TABLE table=test-table [id=90d9a87d10a849099eff548932941858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:28:51.104925 10712 tablet_service.cc:1467] Processing CreateTablet for tablet aaba62d6395045f78fbc04a87d5a06ea (DEFAULT_TABLE table=test-table [id=90d9a87d10a849099eff548932941858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:28:51.106088 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaba62d6395045f78fbc04a87d5a06ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:51.106465 10712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaba62d6395045f78fbc04a87d5a06ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:51.108781 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.078147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60502)
0810 19:28:51.095042 (+ 16895us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.095060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:51.095185 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.097544 (+ 2359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.097550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.097577 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:28:51.100010 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.100026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.102039 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.102044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.104037 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.104052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.107953 (+ 3901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.107959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.108155 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.108191 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.108295 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.108434 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16752,"thread_start_us":9829,"threads_started":1}
I20260810 19:28:51.109051 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.078489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.19.68.1:46651)
0810 19:28:51.096515 (+ 18026us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.096524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.096553 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:28:51.096678 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.096686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.096899 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.097129 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:28:51.100228 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.101865 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.104205 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.107406 (+ 3201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.107539 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.107794 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.108365 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.108540 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17847,"thread_start_us":18813,"threads_started":1}
I20260810 19:28:51.112691 10458 tablet_service.cc:1467] Processing CreateTablet for tablet aaba62d6395045f78fbc04a87d5a06ea (DEFAULT_TABLE table=test-table [id=90d9a87d10a849099eff548932941858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:28:51.114212 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaba62d6395045f78fbc04a87d5a06ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:51.121896 11077 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Bootstrap starting.
I20260810 19:28:51.125620 11077 tablet_bootstrap.cc:654] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:51.127134 11077 log.cc:824] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:51.128845 11079 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Bootstrap starting.
I20260810 19:28:51.131975 11079 tablet_bootstrap.cc:654] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:51.133255 11080 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Bootstrap starting.
I20260810 19:28:51.134650 11079 log.cc:824] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:51.137189 11080 tablet_bootstrap.cc:654] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:51.138645 11080 log.cc:824] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:51.141266 11077 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: No bootstrap required, opened a new log
I20260810 19:28:51.141613 11077 ts_tablet_manager.cc:1397] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20260810 19:28:51.145419 11079 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: No bootstrap required, opened a new log
I20260810 19:28:51.145808 11079 ts_tablet_manager.cc:1397] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260810 19:28:51.150064 11080 tablet_bootstrap.cc:492] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: No bootstrap required, opened a new log
I20260810 19:28:51.150413 11080 ts_tablet_manager.cc:1397] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260810 19:28:51.153295 11077 raft_consensus.cc:348] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.153896 11077 raft_consensus.cc:374] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:51.154047 11077 raft_consensus.cc:729] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b12902f853f40608c87221fec246b7c, State: Initialized, Role: FOLLOWER
I20260810 19:28:51.154844 11077 consensus_queue.cc:260] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.156081 11079 raft_consensus.cc:348] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.156558 11079 raft_consensus.cc:374] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:51.156695 11079 raft_consensus.cc:729] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cff868c18a45cda5c8a4b8070d241b, State: Initialized, Role: FOLLOWER
I20260810 19:28:51.158798 11079 consensus_queue.cc:260] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.161265 11080 raft_consensus.cc:348] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.161779 11080 raft_consensus.cc:374] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:51.161912 11080 raft_consensus.cc:729] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a56328cbc644afb3552a5c3315b064, State: Initialized, Role: FOLLOWER
I20260810 19:28:51.162665 11080 consensus_queue.cc:260] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.163316 11077 ts_tablet_manager.cc:1428] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20260810 19:28:51.167028 11079 ts_tablet_manager.cc:1428] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Time spent starting tablet: real 0.021s user 0.011s sys 0.006s
I20260810 19:28:51.167469 10572 heartbeater.cc:503] Master 127.19.68.62:39351 was elected leader, sending a full tablet report...
I20260810 19:28:51.170471 11030 heartbeater.cc:503] Master 127.19.68.62:39351 was elected leader, sending a full tablet report...
I20260810 19:28:51.171370 11080 ts_tablet_manager.cc:1428] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Time spent starting tablet: real 0.021s user 0.016s sys 0.002s
W20260810 19:28:51.203615 11031 tablet.cc:2369] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:28:51.252401 10818 tablet.cc:2369] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:28:51.270387 10573 tablet.cc:2369] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:28:51.447633 11085 raft_consensus.cc:482] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:28:51.448009 11085 raft_consensus.cc:504] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.450044 11085 leader_election.cc:288] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6a56328cbc644afb3552a5c3315b064 (127.19.68.3:40493), 40cff868c18a45cda5c8a4b8070d241b (127.19.68.1:46651)
I20260810 19:28:51.456573 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.450513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:60909 (local address 127.19.68.1:46651)
0810 19:28:51.450790 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.450796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.451233 (+ 437us) server_negotiation.cc:407] Connection header received
0810 19:28:51.451455 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.451462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.451526 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.451608 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:28:51.452191 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.452968 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.454924 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.455684 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.455811 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.455912 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.456199 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.456323 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 19:28:51.456717 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.450370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.19.68.2:60909)
0810 19:28:51.451109 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.451132 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:51.451298 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.451642 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.451647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.451671 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:28:51.452050 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.452060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.453122 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.453127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.454780 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.454793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.455912 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.455919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.456048 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.456071 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.456164 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.456292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":3,"thread_start_us":409,"threads_started":1}
I20260810 19:28:51.456874 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.450529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:34387 (local address 127.19.68.3:40493)
0810 19:28:51.450773 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.450780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.451453 (+ 673us) server_negotiation.cc:407] Connection header received
0810 19:28:51.451663 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.451668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.451747 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.451839 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:28:51.452449 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.453228 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.455105 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.455938 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.456065 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.456178 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.456475 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.456608 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:28:51.457114 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.450590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:40493 (local address 127.19.68.2:34387)
0810 19:28:51.451342 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.451358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:51.451496 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:51.451875 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:51.451880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:51.451911 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:28:51.452278 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.452289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.453372 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.453377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.454945 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:51.454958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.456169 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.456176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:51.456322 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.456347 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:28:51.456446 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:28:51.456580 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":25,"thread_start_us":451,"threads_started":1}
I20260810 19:28:51.457852 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaba62d6395045f78fbc04a87d5a06ea" candidate_uuid: "1b12902f853f40608c87221fec246b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40cff868c18a45cda5c8a4b8070d241b" is_pre_election: true
I20260810 19:28:51.458240 10958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaba62d6395045f78fbc04a87d5a06ea" candidate_uuid: "1b12902f853f40608c87221fec246b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a56328cbc644afb3552a5c3315b064" is_pre_election: true
I20260810 19:28:51.458351 10496 raft_consensus.cc:2454] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b12902f853f40608c87221fec246b7c in term 0.
I20260810 19:28:51.458736 10958 raft_consensus.cc:2454] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b12902f853f40608c87221fec246b7c in term 0.
I20260810 19:28:51.459110 10637 leader_election.cc:302] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b12902f853f40608c87221fec246b7c, 40cff868c18a45cda5c8a4b8070d241b; no voters:
I20260810 19:28:51.459565 11085 raft_consensus.cc:2781] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:28:51.459726 11085 raft_consensus.cc:482] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:28:51.459847 11085 raft_consensus.cc:3037] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:51.462210 11085 raft_consensus.cc:504] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.462817 11085 leader_election.cc:288] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [CANDIDATE]: Term 1 election: Requested vote from peers e6a56328cbc644afb3552a5c3315b064 (127.19.68.3:40493), 40cff868c18a45cda5c8a4b8070d241b (127.19.68.1:46651)
I20260810 19:28:51.463224 10958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaba62d6395045f78fbc04a87d5a06ea" candidate_uuid: "1b12902f853f40608c87221fec246b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a56328cbc644afb3552a5c3315b064"
I20260810 19:28:51.463341 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaba62d6395045f78fbc04a87d5a06ea" candidate_uuid: "1b12902f853f40608c87221fec246b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40cff868c18a45cda5c8a4b8070d241b"
I20260810 19:28:51.463418 10958 raft_consensus.cc:3037] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:51.463563 10496 raft_consensus.cc:3037] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:51.465673 10958 raft_consensus.cc:2454] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b12902f853f40608c87221fec246b7c in term 1.
I20260810 19:28:51.465921 10496 raft_consensus.cc:2454] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b12902f853f40608c87221fec246b7c in term 1.
I20260810 19:28:51.466058 10631 leader_election.cc:302] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b12902f853f40608c87221fec246b7c, e6a56328cbc644afb3552a5c3315b064; no voters:
I20260810 19:28:51.466332 11085 raft_consensus.cc:2781] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:28:51.466734 11085 raft_consensus.cc:686] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 1 LEADER]: Becoming Leader. State: Replica: 1b12902f853f40608c87221fec246b7c, State: Running, Role: LEADER
I20260810 19:28:51.467185 11085 consensus_queue.cc:237] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } }
I20260810 19:28:51.471807 10235 catalog_manager.cc:5495] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b12902f853f40608c87221fec246b7c (127.19.68.2). New cstate: current_term: 1 leader_uuid: "1b12902f853f40608c87221fec246b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b12902f853f40608c87221fec246b7c" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 } health_report { overall_health: UNKNOWN } } }
I20260810 19:28:51.560160 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.554961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.19.68.2:43853)
0810 19:28:51.555147 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.555151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.555417 (+ 266us) server_negotiation.cc:407] Connection header received
0810 19:28:51.555597 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.555601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.555691 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.555822 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:28:51.556145 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.556886 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.558292 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.558690 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.558785 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.559201 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:28:51.559748 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:28:51.559868 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.559904 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.559968 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:28:51.568130 10958 raft_consensus.cc:1261] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 1 FOLLOWER]: Refusing update from remote peer 1b12902f853f40608c87221fec246b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:28:51.568329 10496 raft_consensus.cc:1261] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 1 FOLLOWER]: Refusing update from remote peer 1b12902f853f40608c87221fec246b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:28:51.569160 11091 consensus_queue.cc:1035] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a56328cbc644afb3552a5c3315b064" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:28:51.569483 11085 consensus_queue.cc:1035] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40cff868c18a45cda5c8a4b8070d241b" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:28:51.584605 11098 mvcc.cc:205] Tried to move back new op lower bound from 7317053978893471744 to 7317053978493345792. Current Snapshot: MvccSnapshot[applied={T|T < 7317053978893471744}]
I20260810 19:28:51.596553 10234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:28:51.596787 10234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:28:51.600019 10234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b913b7efaab4d4ca7420f8909e5ddb6: Sending DeleteTablet for 3 replicas of tablet aaba62d6395045f78fbc04a87d5a06ea
I20260810 19:28:51.601131 10928 tablet_service.cc:1514] Processing DeleteTablet for tablet aaba62d6395045f78fbc04a87d5a06ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:28:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55878
I20260810 19:28:51.601171 10712 tablet_service.cc:1514] Processing DeleteTablet for tablet aaba62d6395045f78fbc04a87d5a06ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:28:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:40218
I20260810 19:28:51.601470 10458 tablet_service.cc:1514] Processing DeleteTablet for tablet aaba62d6395045f78fbc04a87d5a06ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:28:51 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:60502
I20260810 19:28:51.603605 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 10305
I20260810 19:28:51.628473 11116 tablet_replica.cc:330] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: stopping tablet replica
I20260810 19:28:51.630928 11116 raft_consensus.cc:2229] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:28:51.631801 11116 raft_consensus.cc:2258] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:28:51.634311 11114 tablet_replica.cc:330] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: stopping tablet replica
I20260810 19:28:51.634783 11114 raft_consensus.cc:2229] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:28:51.635238 11114 raft_consensus.cc:2258] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:28:51.636040 11117 tablet_replica.cc:330] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: stopping tablet replica
I20260810 19:28:51.636449 11117 raft_consensus.cc:2229] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:28:51.636929 11117 raft_consensus.cc:2258] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:28:51.637706 11114 ts_tablet_manager.cc:1905] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:28:51.639866 11116 ts_tablet_manager.cc:1905] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:28:51.644769 11114 ts_tablet_manager.cc:1918] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:28:51.644959 11114 log.cc:1196] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/aaba62d6395045f78fbc04a87d5a06ea
I20260810 19:28:51.645598 11114 ts_tablet_manager.cc:1939] T aaba62d6395045f78fbc04a87d5a06ea P e6a56328cbc644afb3552a5c3315b064: Deleting consensus metadata
I20260810 19:28:51.647275 11116 ts_tablet_manager.cc:1918] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:28:51.647604 11116 log.cc:1196] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/aaba62d6395045f78fbc04a87d5a06ea
I20260810 19:28:51.647607 10210 catalog_manager.cc:4841] TS e6a56328cbc644afb3552a5c3315b064 (127.19.68.3:40493): tablet aaba62d6395045f78fbc04a87d5a06ea (table test-table [id=90d9a87d10a849099eff548932941858]) successfully deleted
I20260810 19:28:51.648272 11116 ts_tablet_manager.cc:1939] T aaba62d6395045f78fbc04a87d5a06ea P 1b12902f853f40608c87221fec246b7c: Deleting consensus metadata
I20260810 19:28:51.648648 11117 ts_tablet_manager.cc:1905] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:28:51.649777 10211 catalog_manager.cc:4841] TS 1b12902f853f40608c87221fec246b7c (127.19.68.2:43853): tablet aaba62d6395045f78fbc04a87d5a06ea (table test-table [id=90d9a87d10a849099eff548932941858]) successfully deleted
I20260810 19:28:51.656198 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.604363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.19.68.1:46651)
0810 19:28:51.604543 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.604550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.627658 (+ 23108us) server_negotiation.cc:407] Connection header received
0810 19:28:51.627877 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.627882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.628016 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.628118 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:51.630485 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.631583 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.633812 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.634466 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.634586 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.650308 (+ 15722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:51.650321 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:51.650333 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:51.650373 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:51.653277 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.653996 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.654003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.654005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.654146 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.654559 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.654563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.654565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.654949 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.655231 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.655577 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.655803 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:28:51.656507 11117 ts_tablet_manager.cc:1918] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:28:51.656710 11117 log.cc:1196] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/aaba62d6395045f78fbc04a87d5a06ea
I20260810 19:28:51.656855 10558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:51.744779 11117 ts_tablet_manager.cc:1939] T aaba62d6395045f78fbc04a87d5a06ea P 40cff868c18a45cda5c8a4b8070d241b: Deleting consensus metadata
I20260810 19:28:51.744917 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 10305
W20260810 19:28:51.771876 10214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:46651 (error 108)
I20260810 19:28:51.772614 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 10576
I20260810 19:28:51.774376 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.773331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60514)
0810 19:28:51.774072 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.774175 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":512,"threads_started":1}
W20260810 19:28:51.774658 10214 catalog_manager.cc:4584] TS 40cff868c18a45cda5c8a4b8070d241b (127.19.68.1:46651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaba62d6395045f78fbc04a87d5a06ea: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
I20260810 19:28:51.788146 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.773678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.19.68.2:43853)
0810 19:28:51.773998 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.774004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.774023 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:28:51.774083 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.774087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.774164 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.774276 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:51.775757 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.776949 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.778276 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.778981 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.779108 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.781533 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:51.781551 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:51.781569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:51.781614 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:51.785415 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.786081 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.786091 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.786094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.786259 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.786581 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.786586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.786590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.787027 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.787320 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.787538 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.787776 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260810 19:28:51.788795 10809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:51.828540 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.827987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60524)
0810 19:28:51.828231 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.828363 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 19:28:51.829649 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.829240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60526)
0810 19:28:51.829403 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.829492 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 19:28:51.890364 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.889686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60540)
0810 19:28:51.889994 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.890119 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260810 19:28:51.891700 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.891259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60546)
0810 19:28:51.891443 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.891532 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 19:28:51.902045 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 10576
I20260810 19:28:51.931545 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 10823
I20260810 19:28:51.946274 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.932331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.19.68.3:40493)
0810 19:28:51.932544 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:51.932549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:51.932568 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:28:51.932782 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:51.932787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:51.932865 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:51.932958 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:51.934383 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.935143 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.936587 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:51.937008 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:51.937152 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:51.939929 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:51.939942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:51.939955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:51.939991 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:51.943224 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.944169 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.944177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.944179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.944338 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:51.944694 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:51.944696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:51.944698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:51.945145 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:51.945372 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:51.945736 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 19:28:51.945920 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:28:51.946941 11020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:51.972333 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.971859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60560)
0810 19:28:51.972094 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.972182 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 19:28:51.973390 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:51.972939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60574)
0810 19:28:51.973165 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:51.973252 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 19:28:52.034298 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 10823
I20260810 19:28:52.059912 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 10180
I20260810 19:28:52.061124 10291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:52.173861 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:52.173127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:46651 (local address 127.0.0.1:60586)
0810 19:28:52.173564 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:52.173688 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260810 19:28:52.174134 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 10180
I20260810 19:28:52.208105 19728 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:28:52.552811 19728 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:28:52.552992 19728 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:28:52.897200 19728 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:28:52.897 UTC [8540] LOG: received smart shutdown request
2026-08-10 19:28:52.900 UTC [8540] LOG: background worker "logical replication launcher" (PID 8556) exited with exit code 1
2026-08-10 19:28:52.900 UTC [8548] LOG: shutting down
2026-08-10 19:28:53.046 UTC [8540] LOG: database system is shut down
2026-08-10T19:28:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53469 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:28:53 dist-test-slave-2j7r krb5kdc[11143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:28:53 dist-test-slave-2j7r krb5kdc[11143](info): set up 2 sockets
Aug 10 19:28:53 dist-test-slave-2j7r krb5kdc[11143](info): commencing operation
krb5kdc: starting...
W20260810 19:28:54.431783 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:28:54 dist-test-slave-2j7r krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:28:54.709903 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34183
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:28:55.037815 11163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:55.038089 11163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:55.038224 11163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:55.045298 11163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:28:55.045418 11163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:55.045499 11163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:55.045579 11163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:28:55.045641 11163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:28:55.055146 11163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:55.056427 11163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:55.057752 11163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:55.069099 11169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:55.069010 11170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:55.069054 11173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:55.069830 11163 server_base.cc:1034] running on GCE node
I20260810 19:28:55.070565 11163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:55.071863 11163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:55.073064 11163 hybrid_clock.cc:648] HybridClock initialized: now 1786390135073020 us; error 36 us; skew 500 ppm
I20260810 19:28:55.073444 11163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:55.081563 11163 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:28:55.083058 11163 webserver.cc:460] Webserver started at http://127.19.68.62:34029/ using document root <none> and password file <none>
I20260810 19:28:55.083719 11163 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:55.083834 11163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:55.084118 11163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:28:55.087059 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "d2f308c17c8d4005af73f309dc159850"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "22f6f12481fb2ae32a420306defa0492"
server_key_iv: "85f56d9d8ae737bd99ffca842016bdc0"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.087834 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "d2f308c17c8d4005af73f309dc159850"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "22f6f12481fb2ae32a420306defa0492"
server_key_iv: "85f56d9d8ae737bd99ffca842016bdc0"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.092712 11163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:28:55.096114 11185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:55.097430 11163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:28:55.097579 11163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "d2f308c17c8d4005af73f309dc159850"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "22f6f12481fb2ae32a420306defa0492"
server_key_iv: "85f56d9d8ae737bd99ffca842016bdc0"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.097746 11163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:55.117980 11163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:55.123426 11163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:55.123705 11163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:55.141215 11163 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:34183
I20260810 19:28:55.141250 11269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:34183 every 8 connection(s)
I20260810 19:28:55.142555 11163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:28:55.146165 11270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:55.146162 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 11163
I20260810 19:28:55.146399 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:28:55.147094 19728 external_mini_cluster.cc:1428] Setting key 08dcdb0eabd100c90068292cf4d02eb8
I20260810 19:28:55.152503 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:55.175040 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.155557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34183 (local address 127.0.0.1:37278)
0810 19:28:55.156432 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.156494 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:55.158363 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:55.159126 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:55.159132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:55.159822 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:55.160145 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.160162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.162323 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.162327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.163425 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.163438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.164646 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.164651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.164785 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.165416 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:55.165438 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:55.167559 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:55.171225 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.171235 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.171243 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.171688 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.172146 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.172149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.172151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.172321 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.173286 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:55.173294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:55.173488 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 19:28:55.173690 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 19:28:55.174048 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":356,"threads_started":1}
I20260810 19:28:55.175109 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.155556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.19.68.62:34183)
0810 19:28:55.157169 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:55.157180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:55.157236 (+ 56us) server_negotiation.cc:407] Connection header received
0810 19:28:55.158701 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:55.158707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:55.158920 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:55.159094 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:55.160340 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.162154 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.163580 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.164420 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.164585 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.167784 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:55.167794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:55.167808 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:55.167847 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:55.171085 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.171824 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.171834 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.171838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.172029 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.172439 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.172444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.172447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.173152 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:55.173356 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:55.173863 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 19:28:55.174226 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":1372,"spinlock_wait_cycles":13056,"thread_start_us":1179,"threads_started":1}
WARNING: no policy specified for kudu/127.19.68.61@KRBTEST.COM; defaulting to no policy
I20260810 19:28:55.177814 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.61@KRBTEST.COM" created.
I20260810 19:28:55.178879 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.178463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52734)
0810 19:28:55.178657 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.178777 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 19:28:55.179103 11188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
I20260810 19:28:55.180609 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.180285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52748)
0810 19:28:55.180426 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.180498 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 19:28:55.180784 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:42857: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
I20260810 19:28:55.208549 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:42857
--webserver_interface=127.19.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:28:55.230180 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } attempt: 1
I20260810 19:28:55.231516 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.230970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52758)
0810 19:28:55.231204 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.231323 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:28:55.232631 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.232177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52770)
0810 19:28:55.232330 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.232434 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 19:28:55.232870 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:42857: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
I20260810 19:28:55.301167 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } attempt: 2
I20260810 19:28:55.302424 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.301879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52776)
0810 19:28:55.302102 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.302223 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 19:28:55.303546 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.303116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52780)
0810 19:28:55.303276 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.303366 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 19:28:55.303726 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:42857: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
I20260810 19:28:55.394992 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } attempt: 3
I20260810 19:28:55.396281 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.395744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52782)
0810 19:28:55.395980 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.396101 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:28:55.397380 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.396945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52790)
0810 19:28:55.397109 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.397211 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:28:55.397634 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:42857: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
I20260810 19:28:55.540913 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } attempt: 4
I20260810 19:28:55.542240 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.541633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52798)
0810 19:28:55.541952 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.542076 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260810 19:28:55.543285 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.542886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52806)
0810 19:28:55.543031 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.543137 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:28:55.543455 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:42857: Network error: Client connection negotiation failed: client connection to 127.19.68.61:42857: connect: Connection refused (error 111)
W20260810 19:28:55.545670 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:55.545969 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:55.546087 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:28:55.553375 11277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:28:55.553481 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:55.553555 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:55.553632 11277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:28:55.553699 11277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:28:55.563457 11277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:55.564749 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:55.566113 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:55.577631 11289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:55.577546 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:55.577776 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:55.578884 11277 server_base.cc:1034] running on GCE node
I20260810 19:28:55.579596 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:55.580931 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:55.582131 11277 hybrid_clock.cc:648] HybridClock initialized: now 1786390135582068 us; error 53 us; skew 500 ppm
I20260810 19:28:55.582465 11277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:55.585479 11277 init.cc:377] Logged in from keytab as kudu/127.19.68.61@KRBTEST.COM (short username kudu)
I20260810 19:28:55.586975 11277 webserver.cc:460] Webserver started at http://127.19.68.61:33885/ using document root <none> and password file <none>
I20260810 19:28:55.587622 11277 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:55.587745 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:55.588076 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:28:55.591014 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data/instance:
uuid: "cbd4d51456b349608f687cc98563e0cb"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "70cdc1782a56aedc903c303c04fb00b6"
server_key_iv: "f9e06373ea0281df4701b547177bd101"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.591796 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal/instance:
uuid: "cbd4d51456b349608f687cc98563e0cb"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "70cdc1782a56aedc903c303c04fb00b6"
server_key_iv: "f9e06373ea0281df4701b547177bd101"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.596849 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:28:55.600205 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:55.601606 11277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:28:55.601758 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "cbd4d51456b349608f687cc98563e0cb"
format_stamp: "Formatted at 2026-08-10 19:28:55 on dist-test-slave-2j7r"
server_key: "70cdc1782a56aedc903c303c04fb00b6"
server_key_iv: "f9e06373ea0281df4701b547177bd101"
server_key_version: "encryptionkey@0"
I20260810 19:28:55.601950 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:55.618549 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:55.622411 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:55.622668 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:55.640254 11277 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.61:42857
I20260810 19:28:55.640301 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.61:42857 every 8 connection(s)
I20260810 19:28:55.641665 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
I20260810 19:28:55.645534 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:55.647539 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 11277
I20260810 19:28:55.647775 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-1/wal/instance
I20260810 19:28:55.648437 19728 external_mini_cluster.cc:1428] Setting key 5ae7eb52007c84f6ba161a162ed12a9c
I20260810 19:28:55.654558 11397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:55.676234 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.657761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.19.68.62:34183)
0810 19:28:55.657981 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:55.657988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:55.659166 (+ 1178us) server_negotiation.cc:407] Connection header received
0810 19:28:55.661028 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:55.661035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:55.661154 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:55.661267 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:55.663196 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.663944 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.666189 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.666681 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.666799 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.670616 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:55.670621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:55.670624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:55.670658 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:55.673300 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.674005 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.674010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.674014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.674135 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.674538 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.674543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.674545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.674826 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:55.674976 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:55.675684 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 19:28:55.675919 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.19.68.60@KRBTEST.COM; defaulting to no policy
I20260810 19:28:55.677423 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.658171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34183 (local address 127.0.0.1:37292)
0810 19:28:55.658973 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.659026 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:55.660741 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:55.661517 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:55.661528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:55.662241 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:55.662989 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.663012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.664120 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.664125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.666030 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.666052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.666918 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.666925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.667178 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.667874 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:55.667910 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:55.670394 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:55.673432 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.673441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.673456 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.673888 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.674242 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.674248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.674251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.674428 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.674954 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:55.674962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:55.675282 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 19:28:55.675554 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 19:28:55.676051 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.60@KRBTEST.COM" created.
I20260810 19:28:55.679162 11397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:28:55.693920 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.680035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52818)
0810 19:28:55.680270 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.680293 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:55.680392 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:55.681357 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:55.681362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:55.681801 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:55.681945 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.681954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.683344 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.683359 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.684244 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.684257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.685128 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.685135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.685363 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.685876 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:55.685895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:55.688128 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:55.691611 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.691615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.691617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.691978 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.692383 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.692386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.692388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.692507 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.693244 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:55.693251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:55.693383 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 19:28:55.693489 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 19:28:55.693666 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:28:55.694360 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.680194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.19.68.61:42857)
0810 19:28:55.680893 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:55.680899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:55.680924 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:28:55.680968 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:55.680973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:55.681196 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:55.681331 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:55.682112 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.683195 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.684379 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.684959 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.685109 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.688367 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:55.688379 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:55.688392 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:55.688423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:55.691488 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.692116 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.692125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.692128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.692285 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.692616 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.692621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.692624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.693125 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:55.693352 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:55.693657 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 19:28:55.693933 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":385,"threads_started":1}
I20260810 19:28:55.696453 11397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:28:55.697504 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.697038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43116)
0810 19:28:55.697256 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.697368 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260810 19:28:55.697767 11305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.699342 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.699046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43120)
0810 19:28:55.699174 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.699247 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:28:55.699570 11397 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
I20260810 19:28:55.712714 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:41189
--webserver_interface=127.19.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:28:55.748868 11397 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 1
I20260810 19:28:55.750145 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.749660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43134)
0810 19:28:55.749862 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.749982 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 19:28:55.751073 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.750736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43150)
0810 19:28:55.750879 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.750968 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:28:55.751269 11397 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.819466 11397 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 2
I20260810 19:28:55.820664 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.820194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43156)
0810 19:28:55.820422 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.820513 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 19:28:55.821620 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.821294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43158)
0810 19:28:55.821438 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.821504 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:28:55.821820 11397 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.842687 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } attempt: 5
Aug 10 19:28:55 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:28:55.855427 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.843542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.19.68.61:42857)
0810 19:28:55.843735 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:55.843741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:55.843829 (+ 88us) server_negotiation.cc:407] Connection header received
0810 19:28:55.844020 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:55.844025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:55.844111 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:55.844195 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:55.845014 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.845747 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.846926 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.847391 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.847506 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.850472 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:55.850477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:55.850480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:55.850513 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:55.852914 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.853632 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.853636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.853638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.853758 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:55.854112 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:55.854117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:55.854120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:55.854393 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:55.854526 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:55.854910 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 19:28:55.855130 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 19:28:55.855434 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.843511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52830)
0810 19:28:55.843734 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.843754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:55.843899 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:55.844244 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:55.844249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:55.844654 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:55.844846 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.844856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.845880 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.845886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.846765 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:55.846779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:55.847550 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:55.847556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:55.847714 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:55.848278 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:55.848296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:55.850287 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:55.853049 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.853055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.853059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.853520 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.853864 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:55.853869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:55.853874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:55.854007 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:55.854508 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:55.854512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:55.854660 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 19:28:55.854783 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:28:55.855054 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 19:28:55.856622 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:28:55.857607 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.857244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43164)
0810 19:28:55.857394 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.857467 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:28:55.858453 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.858136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43172)
0810 19:28:55.858267 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.858339 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 19:28:55.858636 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.909837 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 1
I20260810 19:28:55.911068 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.910604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43174)
0810 19:28:55.910795 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.910886 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:28:55.912051 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.911666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43180)
0810 19:28:55.911835 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.911941 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 19:28:55.912246 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.913012 11397 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 3
I20260810 19:28:55.913909 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.913511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43182)
0810 19:28:55.913689 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.913769 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:28:55.914911 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.914525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43194)
0810 19:28:55.914734 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.914803 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260810 19:28:55.915100 11397 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
I20260810 19:28:55.980463 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 2
I20260810 19:28:55.981693 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.981217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43208)
0810 19:28:55.981418 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.981510 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:28:55.982645 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:55.982316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43218)
0810 19:28:55.982472 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:55.982541 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:28:55.982862 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
W20260810 19:28:56.052714 11410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:28:56.053056 11410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:28:56.053226 11410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 19:28:56.058292 11397 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 4
I20260810 19:28:56.059370 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.058901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43228)
0810 19:28:56.059124 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.059213 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 19:28:56.060443 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.060120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43238)
0810 19:28:56.060266 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.060332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:28:56.060635 11397 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
W20260810 19:28:56.060855 11410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:28:56.060979 11410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:28:56.061061 11410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:28:56.061170 11410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:28:56.061249 11410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:28:56.071744 11410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:34183,127.19.68.61:42857,127.19.68.60:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:28:56.073132 11410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:28:56.074421 11410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:28:56.085608 11422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:56.085652 11421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:28:56.085686 11424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:28:56.086333 11410 server_base.cc:1034] running on GCE node
I20260810 19:28:56.087073 11410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:28:56.088356 11410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:28:56.089057 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 3
I20260810 19:28:56.089567 11410 hybrid_clock.cc:648] HybridClock initialized: now 1786390136089510 us; error 42 us; skew 500 ppm
I20260810 19:28:56.089903 11410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:28:56.090193 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.089692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43248)
0810 19:28:56.089905 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.090024 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:28:56.091212 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.090785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43260)
0810 19:28:56.090969 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.091076 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 19:28:56.091488 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41189: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41189: connect: Connection refused (error 111)
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390136, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:28:56.093288 11410 init.cc:377] Logged in from keytab as kudu/127.19.68.60@KRBTEST.COM (short username kudu)
I20260810 19:28:56.094691 11410 webserver.cc:460] Webserver started at http://127.19.68.60:46737/ using document root <none> and password file <none>
I20260810 19:28:56.095360 11410 fs_manager.cc:359] Metadata directory not provided
I20260810 19:28:56.095468 11410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:28:56.095758 11410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:28:56.098709 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data/instance:
uuid: "1bdbf5c344a34e12a2ad97efb6680bf5"
format_stamp: "Formatted at 2026-08-10 19:28:56 on dist-test-slave-2j7r"
server_key: "9e19228ca76f9a21c0b973e4bb440c90"
server_key_iv: "b91a3f97c57c81a9b17a03faefe738a1"
server_key_version: "encryptionkey@0"
I20260810 19:28:56.099510 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal/instance:
uuid: "1bdbf5c344a34e12a2ad97efb6680bf5"
format_stamp: "Formatted at 2026-08-10 19:28:56 on dist-test-slave-2j7r"
server_key: "9e19228ca76f9a21c0b973e4bb440c90"
server_key_iv: "b91a3f97c57c81a9b17a03faefe738a1"
server_key_version: "encryptionkey@0"
I20260810 19:28:56.104799 11410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:28:56.108367 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:28:56.109848 11410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:28:56.110026 11410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "1bdbf5c344a34e12a2ad97efb6680bf5"
format_stamp: "Formatted at 2026-08-10 19:28:56 on dist-test-slave-2j7r"
server_key: "9e19228ca76f9a21c0b973e4bb440c90"
server_key_iv: "b91a3f97c57c81a9b17a03faefe738a1"
server_key_version: "encryptionkey@0"
I20260810 19:28:56.110231 11410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:28:56.129061 11410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:28:56.133311 11410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:28:56.133605 11410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:28:56.153072 11410 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.60:41189
I20260810 19:28:56.153115 11525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.60:41189 every 8 connection(s)
I20260810 19:28:56.154485 11410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
I20260810 19:28:56.157842 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 11410
I20260810 19:28:56.158051 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389929706493-19728-0/minicluster-data/master-2/wal/instance
I20260810 19:28:56.158123 11526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:28:56.158587 19728 external_mini_cluster.cc:1428] Setting key b43308a68d45b00bea9359ce916e26ba
I20260810 19:28:56.167629 11526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390136, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:28:56.181241 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.160455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.19.68.60:41189)
0810 19:28:56.161394 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.161410 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.161458 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:28:56.163072 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.163098 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.163592 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.164029 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.165636 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.167483 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.168972 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.169852 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.170034 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.174386 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.174393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.174405 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.174442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.177505 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.178193 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.178202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.178209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.178421 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.178774 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.178788 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.178790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.179261 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.179538 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.179960 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.180237 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":399,"threads_started":1}
I20260810 19:28:56.184183 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.170168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34183 (local address 127.0.0.1:37298)
0810 19:28:56.170942 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.170976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:56.171122 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:56.171479 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:56.171486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:56.172032 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:56.172292 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.172301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.173610 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.173614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.174647 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.174660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.175591 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.175596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.175727 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.176349 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:56.176365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:56.178599 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:56.181742 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.181753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.181760 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.182172 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.182531 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.182536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.182540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.182686 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.183157 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:56.183164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:56.183350 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 19:28:56.183529 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 19:28:56.183785 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":382,"threads_started":1}
I20260810 19:28:56.184298 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.170281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.19.68.62:34183)
0810 19:28:56.170471 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.170479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.171075 (+ 596us) server_negotiation.cc:407] Connection header received
0810 19:28:56.171250 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.171254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.171318 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.171396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.172440 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.173461 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.174797 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.175413 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.175544 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.178768 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.178772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.178775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.178807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.181623 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.182281 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.182287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.182290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.182428 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.182785 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.182789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.182791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.183034 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.183161 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.183675 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.183897 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:28:56.185858 11526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390136, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:28:56.200029 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.186555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:42857 (local address 127.0.0.1:52846)
0810 19:28:56.186720 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.186738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:56.186828 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:56.187133 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:56.187138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:56.187529 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:56.187701 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.187709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.189002 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.189006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.190361 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.190375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.191263 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.191268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.191406 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.191885 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:56.191902 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:56.194258 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:56.197244 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.197249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.197252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.197738 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.198210 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.198216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.198220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.198455 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.199048 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:56.199052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:56.199227 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 19:28:56.199384 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 19:28:56.199689 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 19:28:56.200094 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.183895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.19.68.62:34183)
0810 19:28:56.184614 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.184619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.184635 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:28:56.184673 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.184676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.184725 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.184824 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.186267 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.187487 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.189253 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.189941 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.190083 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.194313 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.194318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.194321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.194353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.197006 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.197887 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.197892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.197895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.198064 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.198536 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.198541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.198544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.198938 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.199205 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.199492 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.199718 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":495,"threads_started":1}
I20260810 19:28:56.200100 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.186583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.19.68.61:42857)
0810 19:28:56.186754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.186759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.186779 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:28:56.186951 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.186955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.187015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.187088 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.187825 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.188838 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.190500 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.191084 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.191219 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.194438 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.194443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.194446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.194485 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.197072 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.197872 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.197877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.197881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.198064 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.198562 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.198567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.198570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.198936 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.199092 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.199501 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.199718 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:28:56.201377 11526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390136, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:28:56.214712 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.202006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43274)
0810 19:28:56.202411 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.202425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:56.202511 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:56.202785 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:56.202790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:56.203245 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:56.203447 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.203459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.204584 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.204587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.205779 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.205794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.206576 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.206582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.206727 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.207262 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:56.207278 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:56.209570 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:56.212377 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.212382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.212386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.212729 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.213040 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.213044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.213048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.213206 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.213734 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:56.213738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:56.213881 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 19:28:56.214006 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 19:28:56.214301 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260810 19:28:56.214921 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.202174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.19.68.60:41189)
0810 19:28:56.202334 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.202339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.202481 (+ 142us) server_negotiation.cc:407] Connection header received
0810 19:28:56.202599 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.202604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.202670 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.202753 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.203575 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.204460 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.205905 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.206419 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.206528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.209728 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.209732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.209735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.209763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.212270 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.212829 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.212835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.212838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.212946 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.213300 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.213305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.213328 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.213628 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.213756 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.214122 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.214531 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":60}
I20260810 19:28:56.217051 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.201977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.19.68.61:42857)
0810 19:28:56.202134 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.202139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.202163 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:28:56.202345 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.202350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.202422 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.202511 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.204001 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.204907 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.206570 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.207035 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.207141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.211459 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.211463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.211466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.211496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.214059 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.214981 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.214987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.214990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.215154 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.215747 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.215752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.215755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.216107 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.216357 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.216635 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.216800 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 19:28:56.220148 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: Bootstrap starting.
I20260810 19:28:56.221550 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:28:56.223356 11526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:56.224396 11526 log.cc:824] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:56.227633 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: No bootstrap required, opened a new log
I20260810 19:28:56.239583 11526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.240046 11526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:56.240172 11526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdbf5c344a34e12a2ad97efb6680bf5, State: Initialized, Role: FOLLOWER
I20260810 19:28:56.240980 11526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.245929 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.226726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.19.68.62:34183)
0810 19:28:56.226917 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.226923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.227588 (+ 665us) server_negotiation.cc:407] Connection header received
0810 19:28:56.227868 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.227875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.227976 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.228050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.230777 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.231985 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.234161 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.234803 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.234915 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.238997 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.239004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.239009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.239052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.242481 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.243454 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.243460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.243463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.243643 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.244093 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.244098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.244102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.244453 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.244689 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.245315 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.245559 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:28:56.246887 11549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.247387 11549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.248195 11526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:28:56.252269 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.226360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.19.68.61:42857)
0810 19:28:56.226525 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.226529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.233694 (+ 7165us) server_negotiation.cc:407] Connection header received
0810 19:28:56.233877 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.233882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.233967 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.234064 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.236441 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.237544 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.239640 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.240260 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.240355 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.243648 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.243655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.243659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.243698 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.247278 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.248720 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.248725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.248728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.250123 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.250653 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.250658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.250662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.251037 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.251257 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.251861 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.252033 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:28:56.254712 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.228056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.19.68.60:41189)
0810 19:28:56.228312 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.228317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.237599 (+ 9282us) server_negotiation.cc:407] Connection header received
0810 19:28:56.238033 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.238039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.238117 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.238194 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.240372 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.241425 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.243834 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.244594 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.244690 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.248255 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.248261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.248265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.248303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.251704 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.252512 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.252517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.252520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.252697 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.253224 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.253229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.253235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.253619 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.253881 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.254259 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.254430 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260810 19:28:56.260419 11569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:28:56.260530 11569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:28:56.268700 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 4
I20260810 19:28:56.258579 11410 master_runner.cc:186] Error getting master registration for 127.19.68.62:34183: OK, instance_id { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" instance_seqno: 1786390135136230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:28:56.269546 11410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
I20260810 19:28:56.282892 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.269263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.19.68.60:41189)
0810 19:28:56.269420 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.269426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.269536 (+ 110us) server_negotiation.cc:407] Connection header received
0810 19:28:56.269715 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.269719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.269793 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.269869 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.270668 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.271701 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.273439 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.273867 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.273955 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.277466 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.277470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.277473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.277501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.280142 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.280976 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.280981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.280985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.281168 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.281623 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.281627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.281629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.281925 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.282038 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.282447 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.282618 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:28:56.283130 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.269217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43296)
0810 19:28:56.269430 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.269451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:56.269570 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:56.269929 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:56.269933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:56.270349 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:56.270529 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.270539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.271824 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.271830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.273302 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.273320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.274008 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.274015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.274212 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.274782 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:56.274818 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:56.277305 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:56.280314 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.280319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.280323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.280826 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.281290 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.281295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.281299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.281492 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.282047 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:56.282052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:56.282221 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 19:28:56.282378 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 19:28:56.282672 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 19:28:56.287599 11270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: Bootstrap starting.
I20260810 19:28:56.291204 11270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:56.292186 11270 log.cc:824] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:56.294991 11270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: No bootstrap required, opened a new log
I20260810 19:28:56.302732 11270 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.303022 11270 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:56.303131 11270 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f308c17c8d4005af73f309dc159850, State: Initialized, Role: FOLLOWER
I20260810 19:28:56.303623 11270 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.304448 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.304833 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.305693 11270 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:28:56.316315 11587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:28:56.316583 11587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:28:56.315501 11163 master_runner.cc:186] Error getting master registration for 127.19.68.61:42857: OK, instance_id { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" instance_seqno: 1786390135635166 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:28:56.324110 11163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:28:56.359846 11397 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } attempt: 5
Aug 10 19:28:56 dist-test-slave-2j7r krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390135, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
W20260810 19:28:56.376508 11195 tablet.cc:2369] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:28:56.376852 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.360541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41189 (local address 127.0.0.1:43306)
0810 19:28:56.360811 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 19:28:56.360832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:28:56.360961 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:28:56.361379 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:28:56.361385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:28:56.361928 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:28:56.362144 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.362157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.363733 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.363738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.365177 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:28:56.365192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.366235 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.366242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:28:56.366442 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.367106 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:28:56.367131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:28:56.369950 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:28:56.373929 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.373936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.373940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.374474 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.374933 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:28:56.374939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:28:56.374943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:28:56.375118 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:28:56.375787 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:28:56.375793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:28:56.375963 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 19:28:56.376125 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:28:56.376405 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 19:28:56.376852 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:28:56.360630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.19.68.60:41189)
0810 19:28:56.360813 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:28:56.360820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:28:56.360924 (+ 104us) server_negotiation.cc:407] Connection header received
0810 19:28:56.361140 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:28:56.361146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:28:56.361229 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:28:56.361327 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:28:56.362366 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.363542 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.365356 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:28:56.366067 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:28:56.366197 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:28:56.370129 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:28:56.370136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:28:56.370141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:28:56.370180 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:28:56.373806 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.374589 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.374595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.374599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.374770 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:28:56.375240 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:28:56.375246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:28:56.375250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:28:56.375659 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:28:56.375858 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 19:28:56.376234 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 19:28:56.376486 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:28:56.382004 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: Bootstrap starting.
W20260810 19:28:56.385809 11443 tablet.cc:2369] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:28:56.386818 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: Neither blocks nor log segments found. Creating new log.
I20260810 19:28:56.387876 11397 log.cc:824] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: Log is configured to *not* fsync() on all Append() calls
I20260810 19:28:56.390558 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: No bootstrap required, opened a new log
I20260810 19:28:56.397632 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.397967 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:28:56.398082 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd4d51456b349608f687cc98563e0cb, State: Initialized, Role: FOLLOWER
I20260810 19:28:56.398651 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.401017 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:28:56.399607 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.404455 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.429553 11277 master_runner.cc:263] No leader master found from master cbd4d51456b349608f687cc98563e0cb
I20260810 19:28:56.429795 11277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 19:28:56.432483 11612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:28:56.432715 11612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 19:28:56.625180 11317 tablet.cc:2369] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:28:56.627930 11549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:28:56.628245 11549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.630362 11549 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2f308c17c8d4005af73f309dc159850 (127.19.68.62:34183), cbd4d51456b349608f687cc98563e0cb (127.19.68.61:42857)
I20260810 19:28:56.631413 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd4d51456b349608f687cc98563e0cb" is_pre_election: true
I20260810 19:28:56.632031 11359 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdbf5c344a34e12a2ad97efb6680bf5 in term 0.
I20260810 19:28:56.633246 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f308c17c8d4005af73f309dc159850" is_pre_election: true
I20260810 19:28:56.633850 11229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdbf5c344a34e12a2ad97efb6680bf5 in term 0.
I20260810 19:28:56.634655 11434 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdbf5c344a34e12a2ad97efb6680bf5, d2f308c17c8d4005af73f309dc159850; no voters:
I20260810 19:28:56.635648 11549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:28:56.635820 11549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:28:56.635915 11549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:56.640167 11549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.641482 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f308c17c8d4005af73f309dc159850"
I20260810 19:28:56.641708 11229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:56.645033 11229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdbf5c344a34e12a2ad97efb6680bf5 in term 1.
I20260810 19:28:56.645684 11434 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdbf5c344a34e12a2ad97efb6680bf5, d2f308c17c8d4005af73f309dc159850; no voters:
I20260810 19:28:56.649451 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd4d51456b349608f687cc98563e0cb"
I20260810 19:28:56.649703 11359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:28:56.653365 11359 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdbf5c344a34e12a2ad97efb6680bf5 in term 1.
I20260810 19:28:56.654281 11549 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [CANDIDATE]: Term 1 election: Requested vote from peers d2f308c17c8d4005af73f309dc159850 (127.19.68.62:34183), cbd4d51456b349608f687cc98563e0cb (127.19.68.61:42857)
I20260810 19:28:56.654736 11549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:28:56.655210 11549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1bdbf5c344a34e12a2ad97efb6680bf5, State: Running, Role: LEADER
I20260810 19:28:56.656064 11549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } }
I20260810 19:28:56.675355 11617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdbf5c344a34e12a2ad97efb6680bf5. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.675941 11617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:56.683328 11619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:28:56.686074 11619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:28:56.707933 11359 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [term 1 FOLLOWER]: Refusing update from remote peer 1bdbf5c344a34e12a2ad97efb6680bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:28:56.709250 11617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:28:56.718166 11229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [term 1 FOLLOWER]: Refusing update from remote peer 1bdbf5c344a34e12a2ad97efb6680bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:28:56.719213 11617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:28:56.758436 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdbf5c344a34e12a2ad97efb6680bf5. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.758831 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.765027 11620 mvcc.cc:205] Tried to move back new op lower bound from 7317053999919632384 to 7317053999756468224. Current Snapshot: MvccSnapshot[applied={T|T < 7317053999919632384}]
I20260810 19:28:56.768441 11624 mvcc.cc:205] Tried to move back new op lower bound from 7317053999919632384 to 7317053999756468224. Current Snapshot: MvccSnapshot[applied={T|T < 7317053999919632384}]
I20260810 19:28:56.769515 11549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.769833 11549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:56.771332 11619 catalog_manager.cc:1347] Generated new cluster ID: 6a49f1a33c1548dcbe427c74ad97ee36
I20260810 19:28:56.771433 11619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:28:56.783959 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.784247 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.784605 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdbf5c344a34e12a2ad97efb6680bf5. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.784868 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.788566 11617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.789438 11617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:28:56.797741 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f308c17c8d4005af73f309dc159850" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34183 } } peers { permanent_uuid: "cbd4d51456b349608f687cc98563e0cb" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 42857 } } peers { permanent_uuid: "1bdbf5c344a34e12a2ad97efb6680bf5" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41189 } } }
I20260810 19:28:56.797995 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:28:56.809257 11619 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:28:56.811161 11619 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:28:56.832835 11619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bdbf5c344a34e12a2ad97efb6680bf5: Generated new TSK 0
I20260810 19:28:56.833707 11619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:28:57.318689 11587 catalog_manager.cc:1259] Loaded cluster ID: 6a49f1a33c1548dcbe427c74ad97ee36
I20260810 19:28:57.318800 11587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: loading cluster ID for follower catalog manager: success
I20260810 19:28:57.321400 11587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: acquiring CA information for follower catalog manager: success
I20260810 19:28:57.323187 11587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2f308c17c8d4005af73f309dc159850: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:28:57.434787 11612 catalog_manager.cc:1259] Loaded cluster ID: 6a49f1a33c1548dcbe427c74ad97ee36
I20260810 19:28:57.434890 11612 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: loading cluster ID for follower catalog manager: success
I20260810 19:28:57.437386 11612 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: acquiring CA information for follower catalog manager: success
I20260810 19:28:57.439157 11612 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbd4d51456b349608f687cc98563e0cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:28:57.926707 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 11163
I20260810 19:28:57.928475 11263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:58.035807 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 11163
I20260810 19:28:58.069355 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 11277
I20260810 19:28:58.070493 11387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:58.177109 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 11277
I20260810 19:28:58.201694 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 11410
I20260810 19:28:58.202826 11518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:28:58.310204 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 11410
2026-08-10T19:28:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5160 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:28:58 dist-test-slave-2j7r krb5kdc[11654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:28:58 dist-test-slave-2j7r krb5kdc[11654](info): set up 2 sockets
Aug 10 19:28:58 dist-test-slave-2j7r krb5kdc[11654](info): commencing operation
krb5kdc: starting...
W20260810 19:28:59.571882 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:28:59 dist-test-slave-2j7r krb5kdc[11654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:28:59.850359 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38957
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:00.180383 11674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:00.180648 11674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:00.180750 11674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:00.187762 11674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:00.187853 11674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:00.187906 11674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:00.187963 11674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:00.188004 11674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:00.197425 11674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:00.198637 11674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:00.199906 11674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:00.211616 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:00.211650 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:00.211683 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:00.212777 11674 server_base.cc:1034] running on GCE node
I20260810 19:29:00.213763 11674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:00.215317 11674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:00.216576 11674 hybrid_clock.cc:648] HybridClock initialized: now 1786390140216513 us; error 44 us; skew 500 ppm
I20260810 19:29:00.216957 11674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:00 dist-test-slave-2j7r krb5kdc[11654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:00.223702 11674 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:00.225327 11674 webserver.cc:460] Webserver started at http://127.19.68.62:40021/ using document root <none> and password file <none>
I20260810 19:29:00.226039 11674 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:00.226166 11674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:00.226495 11674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:00.229630 11674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "f4f3c61c95ca463eb82e014ae60247b5"
format_stamp: "Formatted at 2026-08-10 19:29:00 on dist-test-slave-2j7r"
server_key: "a1af32d14e07d65939cb5491c1032aae"
server_key_iv: "79243744bf7cb0aa393e5d552eb6ecff"
server_key_version: "encryptionkey@0"
I20260810 19:29:00.230381 11674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "f4f3c61c95ca463eb82e014ae60247b5"
format_stamp: "Formatted at 2026-08-10 19:29:00 on dist-test-slave-2j7r"
server_key: "a1af32d14e07d65939cb5491c1032aae"
server_key_iv: "79243744bf7cb0aa393e5d552eb6ecff"
server_key_version: "encryptionkey@0"
I20260810 19:29:00.235610 11674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260810 19:29:00.239144 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:00.240422 11674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:29:00.240579 11674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "f4f3c61c95ca463eb82e014ae60247b5"
format_stamp: "Formatted at 2026-08-10 19:29:00 on dist-test-slave-2j7r"
server_key: "a1af32d14e07d65939cb5491c1032aae"
server_key_iv: "79243744bf7cb0aa393e5d552eb6ecff"
server_key_version: "encryptionkey@0"
I20260810 19:29:00.240754 11674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:00.257660 11674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:00.261524 11674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:00.261781 11674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:00.279171 11674 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:38957
I20260810 19:29:00.279196 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:38957 every 8 connection(s)
I20260810 19:29:00.280493 11674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:00.283977 11808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:00.287509 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 11674
I20260810 19:29:00.287724 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:29:00.288450 19728 external_mini_cluster.cc:1428] Setting key 8b8518fb642dfc7313e17ebbeb290084
I20260810 19:29:00.294513 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5: Bootstrap starting.
I20260810 19:29:00.297956 11808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:00.299142 11808 log.cc:824] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:00.302430 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5: No bootstrap required, opened a new log
Aug 10 19:29:00 dist-test-slave-2j7r krb5kdc[11654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:00.310295 11808 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:29:00.310652 11808 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:00.310750 11808 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f3c61c95ca463eb82e014ae60247b5, State: Initialized, Role: FOLLOWER
I20260810 19:29:00.311486 11808 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:29:00.311784 11808 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:00.311936 11808 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:00.312136 11808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:00.313036 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:00.290250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.19.68.62:38957)
0810 19:29:00.291041 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:00.291052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:00.291092 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:29:00.293241 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:00.293271 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:00.293642 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:00.293948 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:00.295761 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:00.297055 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:00.298988 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:00.299847 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:00.300030 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:00.304688 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:00.304695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:00.304711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:00.304764 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:00.308219 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:00.309036 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:00.309043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:00.309053 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:00.309306 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:00.309739 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:00.309743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:00.309745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:00.310304 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:00.310591 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:00.311104 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 19:29:00.311560 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":361,"threads_started":1}
I20260810 19:29:00.314877 11808 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:29:00.315348 11808 leader_election.cc:302] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f3c61c95ca463eb82e014ae60247b5; no voters:
I20260810 19:29:00.315980 11808 leader_election.cc:288] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:29:00.316172 11814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:00.317194 11814 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [term 1 LEADER]: Becoming Leader. State: Replica: f4f3c61c95ca463eb82e014ae60247b5, State: Running, Role: LEADER
I20260810 19:29:00.317453 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:29:00.317823 11814 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } }
I20260810 19:29:00.318204 11808 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:00.320564 11817 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f3c61c95ca463eb82e014ae60247b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } } }
I20260810 19:29:00.321015 11817 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:00.320472 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f3c61c95ca463eb82e014ae60247b5. Latest consensus state: current_term: 1 leader_uuid: "f4f3c61c95ca463eb82e014ae60247b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3c61c95ca463eb82e014ae60247b5" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38957 } } }
I20260810 19:29:00.322089 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:00.324065 11822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:00.329962 11822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:00.342104 11822 catalog_manager.cc:1347] Generated new cluster ID: e694a19cf7ad4d938ae41786f9a5f044
I20260810 19:29:00.342224 11822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:29:00.362483 11822 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:29:00.363727 11822 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:29:00.374368 11822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f3c61c95ca463eb82e014ae60247b5: Generated new TSK 0
I20260810 19:29:00.375123 11822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:29:00.430878 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 11674
I20260810 19:29:00.432075 11798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:00.528281 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 11674
2026-08-10T19:29:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2213 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:29:00 dist-test-slave-2j7r krb5kdc[11855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:29:00 dist-test-slave-2j7r krb5kdc[11855](info): set up 2 sockets
Aug 10 19:29:00 dist-test-slave-2j7r krb5kdc[11855](info): commencing operation
krb5kdc: starting...
W20260810 19:29:01.792115 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:29:01 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:29:02.078708 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:02.413482 11879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:02.413758 11879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:02.413856 11879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:02.420972 11879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:02.421111 11879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:02.421188 11879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:02.421259 11879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:02.421304 11879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:02.431309 11879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:02.432585 11879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:02.433929 11879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:02.445624 11884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:02.445662 11885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:02.445591 11887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:02.446740 11879 server_base.cc:1034] running on GCE node
I20260810 19:29:02.447494 11879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:02.448839 11879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:02.450048 11879 hybrid_clock.cc:648] HybridClock initialized: now 1786390142449993 us; error 42 us; skew 500 ppm
I20260810 19:29:02.450376 11879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:02 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390142, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:02.453444 11879 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:02.454829 11879 webserver.cc:460] Webserver started at http://127.19.68.62:46371/ using document root <none> and password file <none>
I20260810 19:29:02.455487 11879 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:02.455605 11879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:02.455901 11879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:02.458902 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "fb941d2fd84748abb1234671a3ffc15d"
format_stamp: "Formatted at 2026-08-10 19:29:02 on dist-test-slave-2j7r"
server_key: "f8d265a9eb459135ac491f024448f727"
server_key_iv: "656d321c0cd23431cca4a753484c92a5"
server_key_version: "encryptionkey@0"
I20260810 19:29:02.459643 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "fb941d2fd84748abb1234671a3ffc15d"
format_stamp: "Formatted at 2026-08-10 19:29:02 on dist-test-slave-2j7r"
server_key: "f8d265a9eb459135ac491f024448f727"
server_key_iv: "656d321c0cd23431cca4a753484c92a5"
server_key_version: "encryptionkey@0"
I20260810 19:29:02.464812 11879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:29:02.468307 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:02.469681 11879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:02.469872 11879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "fb941d2fd84748abb1234671a3ffc15d"
format_stamp: "Formatted at 2026-08-10 19:29:02 on dist-test-slave-2j7r"
server_key: "f8d265a9eb459135ac491f024448f727"
server_key_iv: "656d321c0cd23431cca4a753484c92a5"
server_key_version: "encryptionkey@0"
I20260810 19:29:02.470057 11879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:02.501358 11879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:02.505370 11879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:02.505633 11879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:02.523130 11879 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39887
I20260810 19:29:02.523166 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39887 every 8 connection(s)
I20260810 19:29:02.524403 11879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:02.526175 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 11879
I20260810 19:29:02.526412 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:29:02.527315 19728 external_mini_cluster.cc:1428] Setting key d2f84f83c16fbb1f866335286e62dd0d
I20260810 19:29:02.528759 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:02.540472 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap starting.
Aug 10 19:29:02 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:02.544009 12004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:02.545154 12004 log.cc:824] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:02.548636 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: No bootstrap required, opened a new log
I20260810 19:29:02.549787 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:02.529483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.19.68.62:39887)
0810 19:29:02.530239 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:02.530254 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:02.530298 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:29:02.531892 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:02.531914 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:02.532285 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:02.532630 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:02.534117 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:02.535439 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:02.536747 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:02.537448 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:02.537621 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:02.541855 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:02.541863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:02.541880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:02.541920 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:02.544968 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:02.545789 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:02.545802 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:02.545811 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:02.546021 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:02.546510 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:02.546515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:02.546518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:02.547232 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:02.547559 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:02.547925 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 19:29:02.548381 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":338,"threads_started":1}
I20260810 19:29:02.556385 12004 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:02.556720 12004 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:02.556794 12004 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Initialized, Role: FOLLOWER
I20260810 19:29:02.557329 12004 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:02.557546 12004 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:02.557641 12004 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:02.557768 12004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:02.559818 12004 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:02.560253 12004 leader_election.cc:302] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb941d2fd84748abb1234671a3ffc15d; no voters:
I20260810 19:29:02.560781 12004 leader_election.cc:288] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:29:02.560925 12014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:02.561995 12014 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 LEADER]: Becoming Leader. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Running, Role: LEADER
I20260810 19:29:02.562522 12014 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:02.562856 12004 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:02.565150 12016 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb941d2fd84748abb1234671a3ffc15d. Latest consensus state: current_term: 1 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:02.565282 12015 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:02.565588 12016 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:02.565605 12015 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:02.567987 12028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:02.573151 12028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:02.580848 12028 catalog_manager.cc:1347] Generated new cluster ID: 43c88868dcb34424a23137119934dd8b
I20260810 19:29:02.580964 12028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:29:02.640772 12028 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:29:02.642133 12028 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:29:02.653805 12028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Generated new TSK 0
I20260810 19:29:02.654644 12028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:29:02.743721 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:03.074100 12048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:03.074404 12048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:03.074524 12048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:03.081854 12048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:03.081960 12048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:03.082119 12048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:29:03.091848 12048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:03.093240 12048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:03.094659 12048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:03.108022 12056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:03.108029 12055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:03.108060 12058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:03.108861 12048 server_base.cc:1034] running on GCE node
I20260810 19:29:03.109549 12048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:03.110531 12048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:03.111747 12048 hybrid_clock.cc:648] HybridClock initialized: now 1786390143111691 us; error 33 us; skew 500 ppm
I20260810 19:29:03.112047 12048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:03 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:03.115257 12048 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:29:03.116729 12048 webserver.cc:460] Webserver started at http://127.19.68.1:36501/ using document root <none> and password file <none>
I20260810 19:29:03.117456 12048 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:03.117558 12048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:03.117816 12048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:03.120870 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "cbb79cda6b4ede25e59f8d655453747b"
server_key_iv: "d5171aac4033e014adff56c75ca6d748"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.121714 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "cbb79cda6b4ede25e59f8d655453747b"
server_key_iv: "d5171aac4033e014adff56c75ca6d748"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.126870 12048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:29:03.130407 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.131762 12048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:29:03.131917 12048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "cbb79cda6b4ede25e59f8d655453747b"
server_key_iv: "d5171aac4033e014adff56c75ca6d748"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.132107 12048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:03.148728 12048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:03.153129 12048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:03.153425 12048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:03.154335 12048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:03.155633 12048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:03.155718 12048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.155838 12048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:03.155891 12048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.186707 12048 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:44637
I20260810 19:29:03.186749 12269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:44637 every 8 connection(s)
I20260810 19:29:03.188015 12048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:29:03.191408 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 12048
I20260810 19:29:03.191620 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:29:03.192330 19728 external_mini_cluster.cc:1428] Setting key e19db6f04164f40fcfb5a74f7e795e51
Aug 10 19:29:03 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:03.208973 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:03.190520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:38403 (local address 127.19.68.62:39887)
0810 19:29:03.191093 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:03.191099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:03.192050 (+ 951us) server_negotiation.cc:407] Connection header received
0810 19:29:03.193980 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:03.193985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:03.194062 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:03.194181 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:03.196091 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.196818 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.198889 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.199306 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.199414 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:03.203380 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:03.203385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:03.203388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:03.203423 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:03.205833 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:03.206652 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:03.206656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:03.206658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:03.206782 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:03.207221 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:03.207224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:03.207226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:03.207541 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:03.207686 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:03.208363 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 19:29:03.208607 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260810 19:29:03.209760 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:03.191077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:38403)
0810 19:29:03.191874 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:03.191929 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:03.193728 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:03.194431 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:03.194442 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:03.195157 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:03.195906 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:03.195930 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.197001 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.197007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:03.198740 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:03.198756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.199531 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.199537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:03.199811 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:03.200479 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:03.200509 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:03.203147 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:03.205981 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:03.205990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:03.206006 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:03.206552 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:03.206877 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:03.206881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:03.206884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:03.207127 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:03.207656 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:03.207662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:03.207990 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 19:29:03.208215 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:29:03.208629 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":326,"threads_started":1}
I20260810 19:29:03.211303 12270 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:03.211588 12270 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:03.212327 12270 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:03.214522 11925 ts_manager.cc:194] Registered new tserver with Master: 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637)
I20260810 19:29:03.215979 11925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:38403
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:29:03.255836 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:03.586256 12283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:03.586608 12283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:03.586774 12283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:03.594405 12283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:03.594558 12283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:03.594734 12283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:29:03.605077 12283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:03.606585 12283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:03.608026 12283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:03.622044 12289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:03.622004 12295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:03.622072 12291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:03.622820 12283 server_base.cc:1034] running on GCE node
I20260810 19:29:03.623421 12283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:03.624380 12283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:03.625582 12283 hybrid_clock.cc:648] HybridClock initialized: now 1786390143625519 us; error 54 us; skew 500 ppm
I20260810 19:29:03.625895 12283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:03 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:03.629190 12283 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:29:03.630774 12283 webserver.cc:460] Webserver started at http://127.19.68.2:33443/ using document root <none> and password file <none>
I20260810 19:29:03.631479 12283 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:03.631594 12283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:03.631894 12283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:03.634884 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "5a8414bae776432db095180a414299a6"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "1a16f31320ba476121a6ccb721491acd"
server_key_iv: "ffb42da7f83bc62ab255c7bfc387fe6d"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.635700 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "5a8414bae776432db095180a414299a6"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "1a16f31320ba476121a6ccb721491acd"
server_key_iv: "ffb42da7f83bc62ab255c7bfc387fe6d"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.640929 12283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:29:03.644563 12308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.645884 12283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:03.646054 12283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "5a8414bae776432db095180a414299a6"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "1a16f31320ba476121a6ccb721491acd"
server_key_iv: "ffb42da7f83bc62ab255c7bfc387fe6d"
server_key_version: "encryptionkey@0"
I20260810 19:29:03.646230 12283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:03.660291 12283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:03.664031 12283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:03.664289 12283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:03.665199 12283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:03.666494 12283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:03.666589 12283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.666708 12283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:03.666775 12283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:03.697005 12283 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:37611
I20260810 19:29:03.697041 12516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:37611 every 8 connection(s)
I20260810 19:29:03.698274 12283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:29:03.701865 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 12283
I20260810 19:29:03.702055 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:29:03.702695 19728 external_mini_cluster.cc:1428] Setting key 303cd9390a906d4b0b8ce69d0b6330e7
Aug 10 19:29:03 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:03.719681 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:03.700640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:34411 (local address 127.19.68.62:39887)
0810 19:29:03.700834 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:03.700839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:03.702218 (+ 1379us) server_negotiation.cc:407] Connection header received
0810 19:29:03.704580 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:03.704585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:03.704646 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:03.704719 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:03.706842 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.707542 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.709717 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.710145 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.710239 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:03.714272 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:03.714277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:03.714280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:03.714313 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:03.716684 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:03.717444 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:03.717448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:03.717450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:03.717581 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:03.717997 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:03.718001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:03.718003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:03.718276 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:03.718409 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:03.719095 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 19:29:03.719333 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:29:03.720638 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:03.701114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:34411)
0810 19:29:03.702036 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:03.702088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:03.704318 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:03.704994 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:03.705007 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:03.705813 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:03.706644 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:03.706671 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.707722 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.707728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:03.709564 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:03.709582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:03.710387 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:03.710396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:03.710706 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:03.711411 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:03.711447 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:03.714039 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:03.716816 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:03.716825 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:03.716841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:03.717307 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:03.717685 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:03.717690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:03.717694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:03.717882 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:03.718408 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:03.718416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:03.718728 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 19:29:03.718965 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 19:29:03.719496 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260810 19:29:03.722244 12524 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:03.722527 12524 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:03.723229 12524 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:03.724669 11925 ts_manager.cc:194] Registered new tserver with Master: 5a8414bae776432db095180a414299a6 (127.19.68.2:37611)
I20260810 19:29:03.725507 11925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:34411
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:29:03.768819 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:04.096560 12529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:04.096827 12529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:04.096930 12529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:04.104048 12529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:04.104151 12529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:04.104260 12529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:29:04.113775 12529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:04.115015 12529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:04.116549 12529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:04.129747 12542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:04.129722 12548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:04.129958 12541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:04.130503 12529 server_base.cc:1034] running on GCE node
I20260810 19:29:04.131138 12529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:04.132150 12529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:04.133363 12529 hybrid_clock.cc:648] HybridClock initialized: now 1786390144133320 us; error 32 us; skew 500 ppm
I20260810 19:29:04.133668 12529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:04 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:04.136986 12529 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:29:04.138442 12529 webserver.cc:460] Webserver started at http://127.19.68.3:45829/ using document root <none> and password file <none>
I20260810 19:29:04.139103 12529 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:04.139225 12529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:04.139513 12529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:04.142552 12529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "12b67a7443a84aef8a37d298882bbbf1"
format_stamp: "Formatted at 2026-08-10 19:29:04 on dist-test-slave-2j7r"
server_key: "f55fa7a9bf07462e4cff70942f36dec1"
server_key_iv: "b55a7942d15e1b62128b4d1a03d59861"
server_key_version: "encryptionkey@0"
I20260810 19:29:04.143325 12529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "12b67a7443a84aef8a37d298882bbbf1"
format_stamp: "Formatted at 2026-08-10 19:29:04 on dist-test-slave-2j7r"
server_key: "f55fa7a9bf07462e4cff70942f36dec1"
server_key_iv: "b55a7942d15e1b62128b4d1a03d59861"
server_key_version: "encryptionkey@0"
I20260810 19:29:04.148717 12529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 19:29:04.152664 12557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:04.154062 12529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:29:04.154212 12529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "12b67a7443a84aef8a37d298882bbbf1"
format_stamp: "Formatted at 2026-08-10 19:29:04 on dist-test-slave-2j7r"
server_key: "f55fa7a9bf07462e4cff70942f36dec1"
server_key_iv: "b55a7942d15e1b62128b4d1a03d59861"
server_key_version: "encryptionkey@0"
I20260810 19:29:04.154392 12529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:04.168638 12529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:04.172976 12529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:04.173286 12529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:04.174299 12529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:04.175711 12529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:04.175786 12529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:04.175905 12529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:04.175976 12529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:04.206478 12529 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42695
I20260810 19:29:04.206548 12752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42695 every 8 connection(s)
I20260810 19:29:04.207720 12529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:29:04.217360 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 12529
I20260810 19:29:04.217552 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:29:04.218037 19728 external_mini_cluster.cc:1428] Setting key df758d83952d6c0466d55abe051cf4eb
I20260810 19:29:04.218995 12270 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
Aug 10 19:29:04 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:04.227569 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.210306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:52227 (local address 127.19.68.62:39887)
0810 19:29:04.210484 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.210489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.211652 (+ 1163us) server_negotiation.cc:407] Connection header received
0810 19:29:04.213504 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.213511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.213595 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.213683 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:04.215239 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.216011 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.217671 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.218100 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.218218 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.222404 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:04.222408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:04.222412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:04.222447 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:04.224819 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.225573 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.225578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.225580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.225715 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.226113 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.226116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.226118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.226395 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:04.226508 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.227142 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.227316 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:29:04.228565 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.210734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:52227)
0810 19:29:04.211485 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.211522 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.213209 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.213934 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.213942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.214484 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:04.215046 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.215064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.216168 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.216172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.217528 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.217543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.218363 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.218372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.218658 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.219440 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:04.219473 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:04.222199 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:04.224968 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:04.224974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:04.224987 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:04.225463 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:04.225812 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:04.225815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:04.225817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:04.226029 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:04.226487 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:04.226492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:04.226805 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.227042 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.227430 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":279,"threads_started":1}
I20260810 19:29:04.229944 12754 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:04.230227 12754 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:04.231000 12754 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:04.232493 11924 ts_manager.cc:194] Registered new tserver with Master: 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695)
I20260810 19:29:04.233466 11924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:52227
I20260810 19:29:04.233976 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:29:04.251492 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.239144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.19.68.62:39887)
0810 19:29:04.239325 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.239329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.239721 (+ 392us) server_negotiation.cc:407] Connection header received
0810 19:29:04.239828 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.239832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.239891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.239972 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:04.241415 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.242166 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.243520 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.244059 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.244155 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.246593 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:04.246598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:04.246601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:04.246633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:04.249079 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.249764 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.249770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.249773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.249906 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.250210 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.250212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.250214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.250569 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:04.250750 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.251067 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.251269 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:29:04.255797 11924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:29:04.262866 11924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:29:04.285187 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.275501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.19.68.2:37611)
0810 19:29:04.276443 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.276449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.276478 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:29:04.276577 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.276585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.276810 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.276988 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.277677 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.279052 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.282034 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.283534 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.283648 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.283852 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.284608 (+ 756us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.284809 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":697,"threads_started":1}
I20260810 19:29:04.286375 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.275278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37611 (local address 127.0.0.1:37380)
0810 19:29:04.276177 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.276213 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.276388 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.277016 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.277023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.277052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.277464 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.277484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.279231 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.279240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.281814 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.281832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.284077 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.284085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.284320 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.284372 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.285740 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.285979 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":538,"threads_started":1}
I20260810 19:29:04.287189 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.276563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42695 (local address 127.0.0.1:59328)
0810 19:29:04.277964 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.277980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.278104 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.279024 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.279029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.279057 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.279643 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.279655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.281813 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.281819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.284076 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.284093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.286364 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.286371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.286566 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.286601 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.286691 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.286824 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":1111,"threads_started":1}
I20260810 19:29:04.288065 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.276875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.19.68.3:42695)
0810 19:29:04.277681 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.277689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.278237 (+ 548us) server_negotiation.cc:407] Connection header received
0810 19:29:04.278344 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.278351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.278575 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.278748 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.279874 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.281644 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.284289 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.285787 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.285919 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.286184 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.287574 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.287737 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":464,"threads_started":1}
I20260810 19:29:04.291659 12641 tablet_service.cc:1467] Processing CreateTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca (DEFAULT_TABLE table=test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:04.291659 12403 tablet_service.cc:1467] Processing CreateTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca (DEFAULT_TABLE table=test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:04.293190 12641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a08babbb6d4a90a645e8789e90c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:04.293184 12403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a08babbb6d4a90a645e8789e90c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:04.293452 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.278032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:48592)
0810 19:29:04.286776 (+ 8744us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.286793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.286882 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.288031 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.288035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.288063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.288208 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.288219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.289806 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.289811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.290890 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.290917 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.292734 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.292739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.292864 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.292885 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.292965 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.293131 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8651,"thread_start_us":9622,"threads_started":1}
I20260810 19:29:04.293569 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.278568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.19.68.1:44637)
0810 19:29:04.287448 (+ 8880us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.287457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.287486 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:29:04.287580 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.287588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.287824 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.288030 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.288361 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.289663 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.291062 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.292358 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.292491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.292653 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.293036 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.293231 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8700,"spinlock_wait_cycles":22400,"thread_start_us":8677,"threads_started":1}
I20260810 19:29:04.296676 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca (DEFAULT_TABLE table=test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:04.298163 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a08babbb6d4a90a645e8789e90c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:04.308037 12786 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Bootstrap starting.
I20260810 19:29:04.308856 12788 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Bootstrap starting.
I20260810 19:29:04.310443 12790 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Bootstrap starting.
I20260810 19:29:04.312439 12788 tablet_bootstrap.cc:654] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:04.312603 12786 tablet_bootstrap.cc:654] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:04.313689 12790 tablet_bootstrap.cc:654] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:04.313900 12786 log.cc:824] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:04.314322 12788 log.cc:824] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:04.314857 12790 log.cc:824] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:04.316962 12788 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: No bootstrap required, opened a new log
I20260810 19:29:04.317220 12788 ts_tablet_manager.cc:1397] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:29:04.317317 12790 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: No bootstrap required, opened a new log
I20260810 19:29:04.317544 12790 ts_tablet_manager.cc:1397] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 19:29:04.323545 12786 tablet_bootstrap.cc:492] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: No bootstrap required, opened a new log
I20260810 19:29:04.323890 12786 ts_tablet_manager.cc:1397] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260810 19:29:04.325335 12788 raft_consensus.cc:348] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.325917 12788 raft_consensus.cc:374] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:04.325651 12790 raft_consensus.cc:348] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.326081 12790 raft_consensus.cc:374] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:04.326138 12788 raft_consensus.cc:729] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b67a7443a84aef8a37d298882bbbf1, State: Initialized, Role: FOLLOWER
I20260810 19:29:04.326243 12790 raft_consensus.cc:729] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b10999eea8410fa5d5f7d9cdfbd485, State: Initialized, Role: FOLLOWER
I20260810 19:29:04.326877 12788 consensus_queue.cc:260] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.326982 12790 consensus_queue.cc:260] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.327955 12754 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
I20260810 19:29:04.328641 12788 ts_tablet_manager.cc:1428] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 19:29:04.330432 12790 ts_tablet_manager.cc:1428] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260810 19:29:04.333698 12786 raft_consensus.cc:348] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.334153 12786 raft_consensus.cc:374] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:04.334272 12786 raft_consensus.cc:729] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8414bae776432db095180a414299a6, State: Initialized, Role: FOLLOWER
I20260810 19:29:04.334888 12786 consensus_queue.cc:260] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.335963 12524 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
I20260810 19:29:04.336519 12786 ts_tablet_manager.cc:1428] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260810 19:29:04.440253 12273 tablet.cc:2369] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:29:04.450145 12525 tablet.cc:2369] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:29:04.459251 12755 tablet.cc:2369] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:29:04.603932 12797 raft_consensus.cc:482] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:29:04.604252 12797 raft_consensus.cc:504] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.606098 12797 leader_election.cc:288] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8414bae776432db095180a414299a6 (127.19.68.2:37611), 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695)
I20260810 19:29:04.608609 12796 raft_consensus.cc:482] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:29:04.608881 12796 raft_consensus.cc:504] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.610782 12796 leader_election.cc:288] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8414bae776432db095180a414299a6 (127.19.68.2:37611), 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637)
I20260810 19:29:04.614063 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.606504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:48437 (local address 127.19.68.2:37611)
0810 19:29:04.606747 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.606754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.607151 (+ 397us) server_negotiation.cc:407] Connection header received
0810 19:29:04.607416 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.607422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.607507 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.607605 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.608282 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.609300 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.611590 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.612817 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.612940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.613033 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.613621 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.613753 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:29:04.614840 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.607457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:59017 (local address 127.19.68.3:42695)
0810 19:29:04.607630 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.607636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.607939 (+ 303us) server_negotiation.cc:407] Connection header received
0810 19:29:04.608086 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.608092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.608170 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.608254 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.608942 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.610066 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.612422 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.613689 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.613795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.613892 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.614446 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.614574 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 19:29:04.615689 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.607155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42695 (local address 127.19.68.1:59017)
0810 19:29:04.607835 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.607849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.607961 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.608284 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.608288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.608314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.608683 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.608695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.610216 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.610221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.612234 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.612250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.614082 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.614088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.614258 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.614292 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.615227 (+ 935us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.615365 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":470,"threads_started":1}
I20260810 19:29:04.616948 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b67a7443a84aef8a37d298882bbbf1" is_pre_election: true
I20260810 19:29:04.617465 12674 raft_consensus.cc:2454] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b10999eea8410fa5d5f7d9cdfbd485 in term 0.
I20260810 19:29:04.618274 12078 leader_election.cc:302] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b67a7443a84aef8a37d298882bbbf1, 94b10999eea8410fa5d5f7d9cdfbd485; no voters:
I20260810 19:29:04.618819 12797 raft_consensus.cc:2781] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:29:04.619033 12797 raft_consensus.cc:482] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:29:04.619184 12797 raft_consensus.cc:3037] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:04.620958 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.611084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37611 (local address 127.19.68.3:36395)
0810 19:29:04.611555 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.611573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.611676 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.614676 (+ 3000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.614681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.614709 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.614892 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.614904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.616341 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.616345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.618240 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.618270 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.620220 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.620226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.620446 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.620479 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.620567 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.620691 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260810 19:29:04.621850 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.611244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:36395 (local address 127.19.68.2:37611)
0810 19:29:04.614294 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.614300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.614317 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:29:04.614369 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.614374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.614445 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.614520 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.615067 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.616163 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.618764 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.619930 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.620042 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.620132 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.621466 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.621590 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2944,"threads_started":1}
I20260810 19:29:04.622870 12797 raft_consensus.cc:504] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.623106 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.606378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37611 (local address 127.19.68.1:48437)
0810 19:29:04.607035 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.607051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.607216 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.607655 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.607664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.607699 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.608106 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.608120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.609497 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.609501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.611410 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.611425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.613238 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.613244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.613424 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.613458 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.621992 (+ 8534us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.622135 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":325,"threads_started":1}
I20260810 19:29:04.623695 12797 leader_election.cc:288] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [CANDIDATE]: Term 1 election: Requested vote from peers 5a8414bae776432db095180a414299a6 (127.19.68.2:37611), 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695)
I20260810 19:29:04.623843 12440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8414bae776432db095180a414299a6" is_pre_election: true
I20260810 19:29:04.624153 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b67a7443a84aef8a37d298882bbbf1"
I20260810 19:29:04.624368 12674 raft_consensus.cc:3037] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:04.624423 12440 raft_consensus.cc:2454] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b67a7443a84aef8a37d298882bbbf1 in term 0.
I20260810 19:29:04.625574 12440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8414bae776432db095180a414299a6" is_pre_election: true
I20260810 19:29:04.625771 12435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8414bae776432db095180a414299a6"
I20260810 19:29:04.625794 12440 raft_consensus.cc:2454] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b10999eea8410fa5d5f7d9cdfbd485 in term 0.
I20260810 19:29:04.626941 12565 leader_election.cc:302] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b67a7443a84aef8a37d298882bbbf1, 5a8414bae776432db095180a414299a6; no voters:
I20260810 19:29:04.627287 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.612060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:42929 (local address 127.19.68.1:44637)
0810 19:29:04.612253 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.612259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.621266 (+ 9007us) server_negotiation.cc:407] Connection header received
0810 19:29:04.621439 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.621445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.621525 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.621614 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:04.622154 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.623259 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.625246 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.626348 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.626487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.626579 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.626928 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.627052 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:29:04.627271 12674 raft_consensus.cc:2454] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94b10999eea8410fa5d5f7d9cdfbd485 in term 1.
I20260810 19:29:04.627596 12078 leader_election.cc:302] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12b67a7443a84aef8a37d298882bbbf1, 94b10999eea8410fa5d5f7d9cdfbd485; no voters: 5a8414bae776432db095180a414299a6
I20260810 19:29:04.627744 12796 raft_consensus.cc:2741] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 19:29:04.627878 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.611456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.19.68.3:42929)
0810 19:29:04.621182 (+ 9726us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:04.621200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:04.621317 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:04.621771 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:04.621776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:04.621804 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:04.621996 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.622007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.623440 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.623445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.625066 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:04.625080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.626529 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.626535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:04.626739 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.626772 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:29:04.627439 (+ 667us) client_negotiation.cc:240] Negotiation successful
0810 19:29:04.627568 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9613,"thread_start_us":11429,"threads_started":1}
I20260810 19:29:04.628039 12797 raft_consensus.cc:2781] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:04.628535 12797 raft_consensus.cc:686] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 LEADER]: Becoming Leader. State: Replica: 94b10999eea8410fa5d5f7d9cdfbd485, State: Running, Role: LEADER
I20260810 19:29:04.628854 12188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a08babbb6d4a90a645e8789e90c8ca" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" is_pre_election: true
I20260810 19:29:04.629304 12797 consensus_queue.cc:237] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } }
I20260810 19:29:04.633659 11925 catalog_manager.cc:5495] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } health_report { overall_health: HEALTHY } } }
I20260810 19:29:04.651674 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.646365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.19.68.1:44637)
0810 19:29:04.646539 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.646545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.646564 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:29:04.646742 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.646746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.646862 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.646986 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:04.647348 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.648082 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.649546 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.649999 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.650110 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.650602 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:04.651089 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:29:04.651226 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.651333 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.651410 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:29:04.659014 12440 raft_consensus.cc:3037] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:04.659271 12674 raft_consensus.cc:1261] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Refusing update from remote peer 94b10999eea8410fa5d5f7d9cdfbd485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:04.660076 12810 consensus_queue.cc:1035] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:04.662225 12440 raft_consensus.cc:1261] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Refusing update from remote peer 94b10999eea8410fa5d5f7d9cdfbd485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:04.663007 12810 consensus_queue.cc:1035] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:04.687618 11924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:04.687938 11924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:04.691633 11924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Sending DeleteTablet for 3 replicas of tablet 91a08babbb6d4a90a645e8789e90c8ca
I20260810 19:29:04.692868 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 12048
I20260810 19:29:04.692922 12403 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:04 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:37380
I20260810 19:29:04.692950 12641 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:04 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:59328
I20260810 19:29:04.693936 12155 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a08babbb6d4a90a645e8789e90c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:04 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:48592
I20260810 19:29:04.706499 12838 tablet_replica.cc:330] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: stopping tablet replica
I20260810 19:29:04.708026 12838 raft_consensus.cc:2229] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:04.711148 12838 raft_consensus.cc:2258] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:04.714857 12838 ts_tablet_manager.cc:1905] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:04.721282 12841 tablet_replica.cc:330] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: stopping tablet replica
I20260810 19:29:04.722415 12841 raft_consensus.cc:2229] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:29:04.723671 12841 raft_consensus.cc:2258] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:04.725653 12836 tablet_replica.cc:330] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: stopping tablet replica
I20260810 19:29:04.726107 12836 raft_consensus.cc:2229] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:04.726655 12836 raft_consensus.cc:2258] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:04.727327 12841 ts_tablet_manager.cc:1905] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:29:04 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:29:04.730360 12836 ts_tablet_manager.cc:1905] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:04.734679 12841 ts_tablet_manager.cc:1918] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:04.734961 12841 log.cc:1196] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/91a08babbb6d4a90a645e8789e90c8ca
I20260810 19:29:04.735584 12841 ts_tablet_manager.cc:1939] T 91a08babbb6d4a90a645e8789e90c8ca P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting consensus metadata
I20260810 19:29:04.737689 11904 catalog_manager.cc:4841] TS 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637): tablet 91a08babbb6d4a90a645e8789e90c8ca (table test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]) successfully deleted
I20260810 19:29:04.739096 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.693805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.19.68.1:44637)
0810 19:29:04.693996 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.694004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.707294 (+ 13290us) server_negotiation.cc:407] Connection header received
0810 19:29:04.708379 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.708386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.708500 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.708593 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:04.711620 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.712794 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.718830 (+ 6036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.719577 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.719696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.729929 (+ 10233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:04.729945 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:04.729966 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:04.730016 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:04.734383 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.735834 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.735844 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.735848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.736100 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.736908 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.736913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.736917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.737563 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:04.737868 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.738519 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.738751 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:29:04.739985 12263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:04.740355 12838 ts_tablet_manager.cc:1918] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:04.740615 12838 log.cc:1196] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/91a08babbb6d4a90a645e8789e90c8ca
I20260810 19:29:04.741257 12838 ts_tablet_manager.cc:1939] T 91a08babbb6d4a90a645e8789e90c8ca P 5a8414bae776432db095180a414299a6: Deleting consensus metadata
I20260810 19:29:04.742456 12836 ts_tablet_manager.cc:1918] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:04.742740 12836 log.cc:1196] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/91a08babbb6d4a90a645e8789e90c8ca
I20260810 19:29:04.743346 11904 catalog_manager.cc:4841] TS 5a8414bae776432db095180a414299a6 (127.19.68.2:37611): tablet 91a08babbb6d4a90a645e8789e90c8ca (table test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]) successfully deleted
I20260810 19:29:04.743490 12836 ts_tablet_manager.cc:1939] T 91a08babbb6d4a90a645e8789e90c8ca P 12b67a7443a84aef8a37d298882bbbf1: Deleting consensus metadata
I20260810 19:29:04.745067 11904 catalog_manager.cc:4841] TS 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695): tablet 91a08babbb6d4a90a645e8789e90c8ca (table test-table [id=0c3ff3d1f121493f8ba6b7852b3c47bc]) successfully deleted
I20260810 19:29:04.839306 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 12048
I20260810 19:29:04.868203 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 12283
Aug 10 19:29:04 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:29:04.887468 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.869246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.19.68.2:37611)
0810 19:29:04.869467 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.869473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.869501 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:29:04.869666 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.869672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.869771 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.869860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:04.871426 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.872136 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.873532 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:04.874014 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:04.874110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:04.882236 (+ 8126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:04.882253 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:04.882269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:04.882313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:04.885080 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.885744 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.885752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.885754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.885908 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:04.886199 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:04.886202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:04.886204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:04.886559 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:04.886771 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:04.887013 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 19:29:04.887189 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:29:04.888168 12509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:04.975698 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 12283
I20260810 19:29:04.997457 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 12529
Aug 10 19:29:05 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:29:05.012725 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:04.998229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.19.68.3:42695)
0810 19:29:04.998438 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:04.998444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:04.998464 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:29:04.998656 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:04.998660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:04.998728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:04.998826 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:05.000103 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:05.000843 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:05.002158 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:05.002561 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:05.002680 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:05.006896 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:05.006913 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:05.006929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:05.006964 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:05.010026 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:05.010669 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:05.010677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:05.010679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:05.010807 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:05.011171 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:05.011175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:05.011177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:05.011547 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:05.011832 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:05.012145 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 19:29:05.012354 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:29:05.013392 12747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:05.101428 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 12529
I20260810 19:29:05.127890 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 11879
I20260810 19:29:05.129217 11999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:05.241258 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 11879
I20260810 19:29:05.271948 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--webserver_interface=127.19.68.62
--webserver_port=46371
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:05.608974 12850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:05.609356 12850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:05.609512 12850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:05.616840 12850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:05.616957 12850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:05.617048 12850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:05.617161 12850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:05.617244 12850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:05.627244 12850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=46371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:05.628602 12850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:05.630053 12850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:05.654421 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:05.654448 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:05.654472 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:05.655447 12850 server_base.cc:1034] running on GCE node
I20260810 19:29:05.656281 12850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:05.658042 12850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:05.659307 12850 hybrid_clock.cc:648] HybridClock initialized: now 1786390145659216 us; error 51 us; skew 500 ppm
I20260810 19:29:05.659701 12850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:05 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:05.667008 12850 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:05.668926 12850 webserver.cc:460] Webserver started at http://127.19.68.62:46371/ using document root <none> and password file <none>
I20260810 19:29:05.669698 12850 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:05.669826 12850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:05.675892 12850 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:29:05.678995 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:05.680454 12850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:29:05.680605 12850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "fb941d2fd84748abb1234671a3ffc15d"
format_stamp: "Formatted at 2026-08-10 19:29:02 on dist-test-slave-2j7r"
server_key: "f8d265a9eb459135ac491f024448f727"
server_key_iv: "656d321c0cd23431cca4a753484c92a5"
server_key_version: "encryptionkey@0"
I20260810 19:29:05.681705 12850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:05.713876 12850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:05.717911 12850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:05.718150 12850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:05.736562 12850 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39887
I20260810 19:29:05.736598 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39887 every 8 connection(s)
I20260810 19:29:05.738186 12850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:05.745146 12960 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:29:05.748086 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 12850
I20260810 19:29:05.749018 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap starting.
I20260810 19:29:05.749110 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:44637
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=36501
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:29:05.771134 12960 log.cc:824] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:05.783257 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:29:05.783782 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap complete.
I20260810 19:29:05.793007 12960 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:05.793864 12960 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Initialized, Role: FOLLOWER
I20260810 19:29:05.794425 12960 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:05.794639 12960 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:05.794757 12960 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:05.794910 12960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:29:05.799093 12960 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:05.799571 12960 leader_election.cc:302] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb941d2fd84748abb1234671a3ffc15d; no voters:
I20260810 19:29:05.800179 12960 leader_election.cc:288] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:29:05.800374 12969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:29:05.802460 12960 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:05.807312 12969 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 LEADER]: Becoming Leader. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Running, Role: LEADER
I20260810 19:29:05.807917 12969 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:05.812521 12972 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:05.812922 12972 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:05.813303 12973 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb941d2fd84748abb1234671a3ffc15d. Latest consensus state: current_term: 2 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:05.813585 12973 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:05.815963 12994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:05.823694 12994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:05.825037 12994 catalog_manager.cc:1259] Loaded cluster ID: 43c88868dcb34424a23137119934dd8b
I20260810 19:29:05.825145 12994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:29:05.828224 12994 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 19:29:05.845077 12994 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 19:29:05.854063 12994 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:29:05.855778 12994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Loaded TSK: 0
I20260810 19:29:05.857384 12994 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:29:06.088133 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:06.088420 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:06.088550 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:06.095782 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:06.095913 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:06.096061 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:29:06.106187 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:44637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=36501
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:06.107620 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:06.109076 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:06.124259 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:06.124552 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:06.124279 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:06.125681 12963 server_base.cc:1034] running on GCE node
I20260810 19:29:06.126480 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:06.127740 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:06.128990 12963 hybrid_clock.cc:648] HybridClock initialized: now 1786390146128927 us; error 49 us; skew 500 ppm
I20260810 19:29:06.129410 12963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:06 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:06.133474 12963 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:29:06.135231 12963 webserver.cc:460] Webserver started at http://127.19.68.1:36501/ using document root <none> and password file <none>
I20260810 19:29:06.136124 12963 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:06.136289 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:06.142936 12963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:29:06.146245 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.147751 12963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:29:06.147935 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "cbb79cda6b4ede25e59f8d655453747b"
server_key_iv: "d5171aac4033e014adff56c75ca6d748"
server_key_version: "encryptionkey@0"
I20260810 19:29:06.149281 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:06.184311 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:06.189352 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:06.189637 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:06.190761 12963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:06.192420 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:06.192538 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.192646 12963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:06.192745 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.224382 12963 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:44637
I20260810 19:29:06.224479 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:44637 every 8 connection(s)
I20260810 19:29:06.225873 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:29:06.230083 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 12963
I20260810 19:29:06.230707 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:37611
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=33443
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:29:06 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:06.250568 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:06.228885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:49755)
0810 19:29:06.229720 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:06.229767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:06.231588 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:06.233315 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:06.233322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:06.233850 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:06.234424 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:06.234443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.236633 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.236638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:06.238290 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:06.238305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.239570 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.239576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:06.239810 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:06.240504 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:06.240529 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:06.243323 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:06.246659 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:06.246668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:06.246682 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:06.247091 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:06.247524 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:06.247529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:06.247533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:06.247728 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:06.248505 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:06.248513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:06.248826 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 19:29:06.249138 (+ 312us) client_negotiation.cc:240] Negotiation successful
0810 19:29:06.249571 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260810 19:29:06.250998 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:06.228723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:49755 (local address 127.19.68.62:39887)
0810 19:29:06.229361 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:06.229372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:06.229909 (+ 537us) server_negotiation.cc:407] Connection header received
0810 19:29:06.232222 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:06.232244 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:06.232719 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:06.233171 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:06.234679 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.236459 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.238442 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.239362 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.239541 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:06.243601 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:06.243610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:06.243623 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:06.243661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:06.246520 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:06.247232 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:06.247239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:06.247244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:06.247412 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:06.247848 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:06.247852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:06.247854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:06.248362 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:06.248554 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:06.249433 (+ 879us) server_negotiation.cc:299] Negotiation successful
0810 19:29:06.249829 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":252,"threads_started":1}
I20260810 19:29:06.253767 13225 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:06.254099 13225 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:06.254915 13225 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:06.257045 12905 ts_manager.cc:194] Registered new tserver with Master: 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637)
I20260810 19:29:06.258556 12905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:49755
W20260810 19:29:06.559631 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:06.559909 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:06.560045 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:06.566958 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:06.567071 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:06.567221 13231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:29:06.576519 13231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:37611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=33443
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:06.577764 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:06.579072 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:06.591361 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:06.591316 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:06.591285 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:06.597400 13231 server_base.cc:1034] running on GCE node
I20260810 19:29:06.597955 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:06.598832 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:06.600019 13231 hybrid_clock.cc:648] HybridClock initialized: now 1786390146599979 us; error 30 us; skew 500 ppm
I20260810 19:29:06.600301 13231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:06 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:06.603241 13231 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:29:06.604679 13231 webserver.cc:460] Webserver started at http://127.19.68.2:33443/ using document root <none> and password file <none>
I20260810 19:29:06.605382 13231 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:06.605513 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:06.610924 13231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:29:06.613474 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.614694 13231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:29:06.614835 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "5a8414bae776432db095180a414299a6"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "1a16f31320ba476121a6ccb721491acd"
server_key_iv: "ffb42da7f83bc62ab255c7bfc387fe6d"
server_key_version: "encryptionkey@0"
I20260810 19:29:06.615767 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:06.631258 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:06.634994 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:06.635238 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:06.636222 13231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:06.637593 13231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:06.637697 13231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.637804 13231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:06.637878 13231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:06.670022 13231 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:37611
I20260810 19:29:06.670061 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:37611 every 8 connection(s)
I20260810 19:29:06.671622 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:29:06.681391 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 13231
I20260810 19:29:06.682076 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:42695
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=45829
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:29:06 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:06.691390 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:06.674103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:55941 (local address 127.19.68.62:39887)
0810 19:29:06.674295 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:06.674300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:06.675441 (+ 1141us) server_negotiation.cc:407] Connection header received
0810 19:29:06.677239 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:06.677244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:06.677333 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:06.677425 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:06.678903 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.679577 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.681217 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.681670 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.681767 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:06.685661 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:06.685666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:06.685669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:06.685700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:06.688524 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:06.689204 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:06.689209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:06.689212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:06.689377 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:06.689762 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:06.689767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:06.689770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:06.690089 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:06.690234 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:06.690845 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 19:29:06.691066 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:29:06.692147 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:06.674529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:55941)
0810 19:29:06.675245 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:06.675323 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:06.676947 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:06.677671 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:06.677678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:06.678189 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:06.678736 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:06.678754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.679735 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.679739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:06.681050 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:06.681066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:06.681889 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:06.681898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:06.682177 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:06.682829 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:06.682855 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:06.685474 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:06.688663 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:06.688669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:06.688680 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:06.689066 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:06.689473 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:06.689476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:06.689478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:06.689656 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:06.690199 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:06.690205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:06.690492 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 19:29:06.690700 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 19:29:06.691058 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":286,"threads_started":1}
I20260810 19:29:06.693442 13446 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:06.693706 13446 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:06.694476 13446 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:06.695994 12905 ts_manager.cc:194] Registered new tserver with Master: 5a8414bae776432db095180a414299a6 (127.19.68.2:37611)
I20260810 19:29:06.696772 12905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:55941
W20260810 19:29:07.036275 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:07.036545 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:07.036680 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:07.043784 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:07.043902 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:07.044054 13449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:29:07.053433 13449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:42695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=45829
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:07.054736 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:07.056402 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:07.069299 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:07.069329 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:07.069224 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:07.070274 13449 server_base.cc:1034] running on GCE node
I20260810 19:29:07.070860 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:07.071777 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:07.072970 13449 hybrid_clock.cc:648] HybridClock initialized: now 1786390147072926 us; error 37 us; skew 500 ppm
I20260810 19:29:07.073309 13449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:07 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390147, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:07.076253 13449 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:29:07.077756 13449 webserver.cc:460] Webserver started at http://127.19.68.3:45829/ using document root <none> and password file <none>
I20260810 19:29:07.078383 13449 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:07.078511 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:07.083787 13449 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:29:07.086584 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:07.087895 13449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:07.088037 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "12b67a7443a84aef8a37d298882bbbf1"
format_stamp: "Formatted at 2026-08-10 19:29:04 on dist-test-slave-2j7r"
server_key: "f55fa7a9bf07462e4cff70942f36dec1"
server_key_iv: "b55a7942d15e1b62128b4d1a03d59861"
server_key_version: "encryptionkey@0"
I20260810 19:29:07.088979 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:07.105914 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:07.110265 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:07.110540 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:07.111621 13449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:07.113160 13449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:07.113267 13449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:07.113373 13449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:07.113438 13449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:07.145113 13449 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42695
I20260810 19:29:07.145241 13670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42695 every 8 connection(s)
I20260810 19:29:07.146546 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:29:07.149058 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 13449
Aug 10 19:29:07 dist-test-slave-2j7r krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390147, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:07.168622 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.149214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:58113 (local address 127.19.68.62:39887)
0810 19:29:07.149434 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.149441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.150746 (+ 1305us) server_negotiation.cc:407] Connection header received
0810 19:29:07.152586 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.152591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.152673 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.152759 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:07.154356 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.155216 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.156753 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.157357 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.157478 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.162256 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:07.162261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:07.162265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:07.162297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:07.165493 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.166185 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.166189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.166191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.166320 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.166744 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.166747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.166750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.167162 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:07.167369 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.168062 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.168303 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:29:07.169598 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.149708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:58113)
0810 19:29:07.150584 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.150638 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.152318 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.153000 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.153008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.153575 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:07.154143 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.154161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.155425 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.155428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.156617 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.156629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.157756 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.157763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.158065 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.159006 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:07.159070 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:07.162054 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:07.165619 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:07.165628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:07.165644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:07.166062 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:07.166430 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:07.166435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:07.166439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:07.166629 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:07.167350 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:07.167358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:07.167680 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.167916 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.168417 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260810 19:29:07.171070 13672 heartbeater.cc:347] Connected to a master server at 127.19.68.62:39887
I20260810 19:29:07.171404 13672 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:07.172281 13672 heartbeater.cc:511] Master 127.19.68.62:39887 requested a full tablet report, sending...
I20260810 19:29:07.173951 12905 ts_manager.cc:194] Registered new tserver with Master: 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695)
I20260810 19:29:07.174742 12905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:58113
I20260810 19:29:07.176455 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.150031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.19.68.62:39887)
0810 19:29:07.163012 (+ 12981us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.163016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.163031 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:29:07.163071 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.163074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.163125 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.163209 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:07.165274 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.166078 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.167514 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.167908 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.168011 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.170890 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:07.170895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:07.170898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:07.170926 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:07.173919 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.174565 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.174569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.174571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.174700 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.175088 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.175092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.175094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.175352 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:07.175606 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.175876 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.176091 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12889,"thread_start_us":12752,"threads_started":1}
I20260810 19:29:07.178784 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:29:07.185035 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.179600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.19.68.62:39887)
0810 19:29:07.179760 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.179765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.180190 (+ 425us) server_negotiation.cc:407] Connection header received
0810 19:29:07.180392 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.180396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.180518 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.180636 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:07.181019 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.181702 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.183128 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.183524 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.183634 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.184087 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:07.184498 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:29:07.184620 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.184729 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.184812 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:29:07.185882 12905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:29:07.193260 12905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:29:07.225811 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.206274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.19.68.1:44637)
0810 19:29:07.206921 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.206932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.213664 (+ 6732us) server_negotiation.cc:407] Connection header received
0810 19:29:07.214033 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.214044 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.214342 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.214530 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:07.215598 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.218586 (+ 2988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.222134 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.223857 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.224067 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.224293 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.225129 (+ 836us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.225361 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":321,"threads_started":1}
I20260810 19:29:07.227095 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.205902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39026)
0810 19:29:07.213541 (+ 7639us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.213560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.213787 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.214906 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.214915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.214955 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:07.215363 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.215375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.218795 (+ 3420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.218801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.221268 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.221284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.224503 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.224510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.224773 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.224835 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.226363 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.226637 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7522,"mutex_wait_us":5,"thread_start_us":7389,"threads_started":1}
I20260810 19:29:07.229851 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.204670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:42695 (local address 127.0.0.1:50070)
0810 19:29:07.205581 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.205626 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.205801 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.222462 (+ 16661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.222467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.222492 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:07.222884 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.222898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.224649 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.224654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.226852 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.226871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.229043 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.229050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.229252 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.229287 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.229378 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.229518 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":457,"threads_started":1}
I20260810 19:29:07.229991 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.204826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.19.68.3:42695)
0810 19:29:07.221902 (+ 17076us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.221909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.221939 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:29:07.222032 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.222040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.222268 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.222463 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:07.223096 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.224465 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.227072 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.228510 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.228640 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.228885 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.229462 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.229627 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16878,"thread_start_us":16727,"threads_started":1}
I20260810 19:29:07.234591 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 (DEFAULT_TABLE table=test-table [id=2f0902cda1c042ee975f9b121cd356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:07.234591 13112 tablet_service.cc:1467] Processing CreateTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 (DEFAULT_TABLE table=test-table [id=2f0902cda1c042ee975f9b121cd356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:07.236146 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339862c5a1e84addb7125c44e3dfd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:07.236163 13112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339862c5a1e84addb7125c44e3dfd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:07.236254 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.213989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.19.68.2:37611)
0810 19:29:07.229156 (+ 15167us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.229165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.229192 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:29:07.229272 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.229277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.229538 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.229739 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:07.230585 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.232038 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.233730 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.234818 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.234982 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.235164 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.235781 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.235955 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14993,"thread_start_us":14889,"threads_started":1}
I20260810 19:29:07.236192 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.213674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37611 (local address 127.0.0.1:50104)
0810 19:29:07.220431 (+ 6757us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.220450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.220554 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.229989 (+ 9435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.229993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.230022 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:07.230382 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.230406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.232174 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.232180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.233557 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.233570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.235423 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.235429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.235601 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.235631 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.235714 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.235852 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":6948,"threads_started":1}
I20260810 19:29:07.240147 13333 tablet_service.cc:1467] Processing CreateTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 (DEFAULT_TABLE table=test-table [id=2f0902cda1c042ee975f9b121cd356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:07.241367 13333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339862c5a1e84addb7125c44e3dfd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:07.252082 13698 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Bootstrap starting.
I20260810 19:29:07.252878 13700 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Bootstrap starting.
I20260810 19:29:07.256136 13698 tablet_bootstrap.cc:654] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:07.257157 13698 log.cc:824] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:07.257601 13700 tablet_bootstrap.cc:654] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:07.257778 13701 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Bootstrap starting.
I20260810 19:29:07.259434 13700 log.cc:824] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:07.260936 13698 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: No bootstrap required, opened a new log
I20260810 19:29:07.261047 13701 tablet_bootstrap.cc:654] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:07.261248 13698 ts_tablet_manager.cc:1397] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260810 19:29:07.261350 13225 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
I20260810 19:29:07.262283 13701 log.cc:824] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:07.262445 13700 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: No bootstrap required, opened a new log
I20260810 19:29:07.262706 13700 ts_tablet_manager.cc:1397] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260810 19:29:07.272035 13700 raft_consensus.cc:348] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.272425 13700 raft_consensus.cc:374] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:07.272287 13698 raft_consensus.cc:348] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.272739 13698 raft_consensus.cc:374] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:07.272833 13698 raft_consensus.cc:729] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b67a7443a84aef8a37d298882bbbf1, State: Initialized, Role: FOLLOWER
I20260810 19:29:07.273470 13700 raft_consensus.cc:729] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b10999eea8410fa5d5f7d9cdfbd485, State: Initialized, Role: FOLLOWER
I20260810 19:29:07.273427 13698 consensus_queue.cc:260] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.274549 13672 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
I20260810 19:29:07.275224 13698 ts_tablet_manager.cc:1428] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 19:29:07.275327 13701 tablet_bootstrap.cc:492] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: No bootstrap required, opened a new log
I20260810 19:29:07.275583 13701 ts_tablet_manager.cc:1397] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260810 19:29:07.274207 13700 consensus_queue.cc:260] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.278659 13700 ts_tablet_manager.cc:1428] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Time spent starting tablet: real 0.015s user 0.009s sys 0.002s
I20260810 19:29:07.283936 13701 raft_consensus.cc:348] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.284269 13701 raft_consensus.cc:374] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:07.284370 13701 raft_consensus.cc:729] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8414bae776432db095180a414299a6, State: Initialized, Role: FOLLOWER
I20260810 19:29:07.284991 13701 consensus_queue.cc:260] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.286119 13446 heartbeater.cc:503] Master 127.19.68.62:39887 was elected leader, sending a full tablet report...
I20260810 19:29:07.286752 13701 ts_tablet_manager.cc:1428] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260810 19:29:07.398099 13673 tablet.cc:2369] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:29:07.423610 13447 tablet.cc:2369] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:29:07.435216 13707 raft_consensus.cc:482] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:29:07.435535 13707 raft_consensus.cc:504] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.437451 13707 leader_election.cc:288] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637), 5a8414bae776432db095180a414299a6 (127.19.68.2:37611)
I20260810 19:29:07.443879 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.437829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:55073 (local address 127.19.68.1:44637)
0810 19:29:07.438059 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.438066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.438121 (+ 55us) server_negotiation.cc:407] Connection header received
0810 19:29:07.438340 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.438346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.438447 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.438546 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:07.439036 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.439744 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.441796 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.443019 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.443163 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.443257 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.443578 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.443676 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:29:07.443967 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.437835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.19.68.3:55073)
0810 19:29:07.438035 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.438052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.438176 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.438593 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.438598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.438629 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:07.438882 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.438895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.439885 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.439890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.441624 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.441637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.443214 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.443221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.443409 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.443447 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.443539 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.443679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:29:07.445019 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339862c5a1e84addb7125c44e3dfd1c6" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" is_pre_election: true
I20260810 19:29:07.445325 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.438446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:39373 (local address 127.19.68.2:37611)
0810 19:29:07.438639 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.438644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.439212 (+ 568us) server_negotiation.cc:407] Connection header received
0810 19:29:07.439370 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.439374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.439431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.439502 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:07.440075 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.440940 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.443227 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.444424 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.444526 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.444594 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.444965 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.445062 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:29:07.445441 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.438285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37611 (local address 127.19.68.3:39373)
0810 19:29:07.439113 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.439126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:07.439245 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:07.439534 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:07.439538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:07.439563 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:07.439860 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.439871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.441110 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.441116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.443058 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:07.443074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.444620 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.444627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:07.444790 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.444824 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:29:07.444925 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 19:29:07.445053 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":514,"threads_started":1}
I20260810 19:29:07.445710 13157 raft_consensus.cc:2454] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b67a7443a84aef8a37d298882bbbf1 in term 0.
I20260810 19:29:07.446434 13487 leader_election.cc:302] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b67a7443a84aef8a37d298882bbbf1, 94b10999eea8410fa5d5f7d9cdfbd485; no voters:
I20260810 19:29:07.446483 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339862c5a1e84addb7125c44e3dfd1c6" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8414bae776432db095180a414299a6" is_pre_election: true
I20260810 19:29:07.446882 13707 raft_consensus.cc:2781] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:29:07.447010 13707 raft_consensus.cc:482] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:29:07.446977 13372 raft_consensus.cc:2454] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b67a7443a84aef8a37d298882bbbf1 in term 0.
I20260810 19:29:07.447106 13707 raft_consensus.cc:3037] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:07.449471 13707 raft_consensus.cc:504] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.450130 13707 leader_election.cc:288] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 election: Requested vote from peers 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637), 5a8414bae776432db095180a414299a6 (127.19.68.2:37611)
I20260810 19:29:07.450614 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339862c5a1e84addb7125c44e3dfd1c6" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
I20260810 19:29:07.450634 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339862c5a1e84addb7125c44e3dfd1c6" candidate_uuid: "12b67a7443a84aef8a37d298882bbbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8414bae776432db095180a414299a6"
I20260810 19:29:07.450804 13157 raft_consensus.cc:3037] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:07.450827 13372 raft_consensus.cc:3037] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:07.452971 13157 raft_consensus.cc:2454] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b67a7443a84aef8a37d298882bbbf1 in term 1.
I20260810 19:29:07.453222 13372 raft_consensus.cc:2454] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b67a7443a84aef8a37d298882bbbf1 in term 1.
I20260810 19:29:07.453410 13487 leader_election.cc:302] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b67a7443a84aef8a37d298882bbbf1, 94b10999eea8410fa5d5f7d9cdfbd485; no voters:
I20260810 19:29:07.453717 13707 raft_consensus.cc:2781] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:07.454207 13707 raft_consensus.cc:686] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 1 LEADER]: Becoming Leader. State: Replica: 12b67a7443a84aef8a37d298882bbbf1, State: Running, Role: LEADER
I20260810 19:29:07.454670 13707 consensus_queue.cc:237] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } }
I20260810 19:29:07.459546 12904 catalog_manager.cc:5495] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3). New cstate: current_term: 1 leader_uuid: "12b67a7443a84aef8a37d298882bbbf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b67a7443a84aef8a37d298882bbbf1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 42695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 } health_report { overall_health: UNKNOWN } } }
W20260810 19:29:07.478181 13226 tablet.cc:2369] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:29:07.511026 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.505522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.19.68.3:42695)
0810 19:29:07.505719 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.505725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.505745 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:29:07.506001 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.506005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.506092 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.506224 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:07.506617 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.507368 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.508859 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.509292 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.509407 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.509946 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:07.510443 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:29:07.510632 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.510683 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.510776 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:29:07.519192 13157 raft_consensus.cc:1261] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Refusing update from remote peer 12b67a7443a84aef8a37d298882bbbf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:07.519192 13372 raft_consensus.cc:1261] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Refusing update from remote peer 12b67a7443a84aef8a37d298882bbbf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:07.520203 13711 consensus_queue.cc:1035] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94b10999eea8410fa5d5f7d9cdfbd485" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:07.520560 13707 consensus_queue.cc:1035] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8414bae776432db095180a414299a6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:07.545053 12905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:07.545349 12905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:07.548892 12905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Sending DeleteTablet for 3 replicas of tablet 339862c5a1e84addb7125c44e3dfd1c6
I20260810 19:29:07.550266 13333 tablet_service.cc:1514] Processing DeleteTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:07 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:50104
I20260810 19:29:07.550227 13112 tablet_service.cc:1514] Processing DeleteTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:07 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:39026
I20260810 19:29:07.551029 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 339862c5a1e84addb7125c44e3dfd1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:07 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:50070
I20260810 19:29:07.551184 13733 tablet_replica.cc:330] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: stopping tablet replica
I20260810 19:29:07.551635 13733 raft_consensus.cc:2229] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:07.552111 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 12963
I20260810 19:29:07.552134 13733 raft_consensus.cc:2258] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:07.555603 13733 ts_tablet_manager.cc:1905] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:07.560561 13735 tablet_replica.cc:330] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: stopping tablet replica
I20260810 19:29:07.562119 13736 tablet_replica.cc:330] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: stopping tablet replica
I20260810 19:29:07.562696 13736 raft_consensus.cc:2229] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:29:07.563482 13736 raft_consensus.cc:2258] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:07.561069 13735 raft_consensus.cc:2229] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:07.564069 13735 raft_consensus.cc:2258] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:07.565850 13736 ts_tablet_manager.cc:1905] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:07.566722 13735 ts_tablet_manager.cc:1905] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:07.570724 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.553415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.19.68.1:44637)
0810 19:29:07.553613 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.553618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.553638 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:29:07.553694 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.553699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.553804 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.553914 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:07.556009 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.557213 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.559163 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.559834 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.559962 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.563408 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:07.563425 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:07.563441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:07.563490 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:07.567428 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.568221 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.568230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.568233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.568430 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.568824 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.568828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.568831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.569453 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:07.569754 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.570082 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.570315 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:29:07.571828 13217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:07.572968 13736 ts_tablet_manager.cc:1918] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:07.573249 13736 log.cc:1196] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/339862c5a1e84addb7125c44e3dfd1c6
I20260810 19:29:07.573701 13736 ts_tablet_manager.cc:1939] T 339862c5a1e84addb7125c44e3dfd1c6 P 12b67a7443a84aef8a37d298882bbbf1: Deleting consensus metadata
I20260810 19:29:07.574160 13733 ts_tablet_manager.cc:1918] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:07.574354 13733 log.cc:1196] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/339862c5a1e84addb7125c44e3dfd1c6
I20260810 19:29:07.574892 13733 ts_tablet_manager.cc:1939] T 339862c5a1e84addb7125c44e3dfd1c6 P 5a8414bae776432db095180a414299a6: Deleting consensus metadata
I20260810 19:29:07.575209 12885 catalog_manager.cc:4841] TS 12b67a7443a84aef8a37d298882bbbf1 (127.19.68.3:42695): tablet 339862c5a1e84addb7125c44e3dfd1c6 (table test-table [id=2f0902cda1c042ee975f9b121cd356eb]) successfully deleted
I20260810 19:29:07.576738 12885 catalog_manager.cc:4841] TS 5a8414bae776432db095180a414299a6 (127.19.68.2:37611): tablet 339862c5a1e84addb7125c44e3dfd1c6 (table test-table [id=2f0902cda1c042ee975f9b121cd356eb]) successfully deleted
I20260810 19:29:07.664201 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 12963
W20260810 19:29:07.690459 12885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:44637 (error 108)
I20260810 19:29:07.691433 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 13231
I20260810 19:29:07.692701 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.692153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39048)
0810 19:29:07.692403 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.692533 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260810 19:29:07.693560 12885 catalog_manager.cc:4584] TS 94b10999eea8410fa5d5f7d9cdfbd485 (127.19.68.1:44637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 339862c5a1e84addb7125c44e3dfd1c6: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
I20260810 19:29:07.706300 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.692602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.19.68.2:37611)
0810 19:29:07.692966 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.692972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.692993 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:29:07.693057 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.693063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.693182 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.693290 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:07.694861 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.695628 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.696942 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.697410 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.697518 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.700395 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:07.700408 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:07.700421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:07.700458 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:07.703378 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.704128 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.704135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.704137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.704278 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.704669 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.704673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.704675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.705054 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:07.705370 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.705742 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.705970 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260810 19:29:07.706936 13440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:07.747252 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.746644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39050)
0810 19:29:07.746956 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.747071 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260810 19:29:07.748668 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.748197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39056)
0810 19:29:07.748415 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.748503 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 19:29:07.809748 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.809187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39070)
0810 19:29:07.809425 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.809532 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 19:29:07.811015 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.810589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39084)
0810 19:29:07.810799 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.810881 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 19:29:07.812448 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 13231
I20260810 19:29:07.838167 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 13449
I20260810 19:29:07.852592 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.839119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.19.68.3:42695)
0810 19:29:07.839316 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:07.839322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:07.839347 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:29:07.839495 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:07.839499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:07.839570 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:07.839661 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:07.841108 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.841950 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.843249 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:07.843685 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:07.843806 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:07.846517 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:07.846532 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:07.846547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:07.846594 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:07.849537 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.850235 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.850263 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.850266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.850469 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:07.850863 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:07.850868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:07.850871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:07.851290 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:07.851635 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:07.851957 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 19:29:07.852182 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:29:07.853426 13663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:07.892089 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.891471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39094)
0810 19:29:07.891722 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.891853 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 19:29:07.893254 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:07.892772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39096)
0810 19:29:07.893005 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:07.893120 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 19:29:07.976711 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 13449
I20260810 19:29:07.999775 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 12850
I20260810 19:29:08.000871 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:08.112892 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 12850
I20260810 19:29:08.112969 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:08.112414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:44637 (local address 127.0.0.1:39104)
0810 19:29:08.112605 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:08.112759 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:29:08.143883 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--webserver_interface=127.19.68.62
--webserver_port=46371
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:08.478196 13745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:08.478488 13745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:08.478610 13745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:08.485775 13745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:08.485879 13745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:08.485970 13745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:08.486049 13745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:08.486107 13745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:08.496263 13745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=46371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:08.497572 13745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:08.498898 13745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:08.510317 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:08.510385 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:08.511019 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:08.511142 13745 server_base.cc:1034] running on GCE node
I20260810 19:29:08.511911 13745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:08.513219 13745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:08.514472 13745 hybrid_clock.cc:648] HybridClock initialized: now 1786390148514413 us; error 41 us; skew 500 ppm
I20260810 19:29:08.514824 13745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:08 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390148, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:08.517822 13745 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:08.519222 13745 webserver.cc:460] Webserver started at http://127.19.68.62:46371/ using document root <none> and password file <none>
I20260810 19:29:08.519968 13745 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:08.520076 13745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:08.525794 13745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:29:08.528779 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:08.530395 13745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:29:08.530565 13745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "fb941d2fd84748abb1234671a3ffc15d"
format_stamp: "Formatted at 2026-08-10 19:29:02 on dist-test-slave-2j7r"
server_key: "f8d265a9eb459135ac491f024448f727"
server_key_iv: "656d321c0cd23431cca4a753484c92a5"
server_key_version: "encryptionkey@0"
I20260810 19:29:08.531549 13745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:08.555187 13745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:08.559171 13745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:08.559429 13745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:08.577190 13745 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:39887
I20260810 19:29:08.577210 13850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:39887 every 8 connection(s)
I20260810 19:29:08.578670 13745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:08.580789 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 13745
I20260810 19:29:08.581820 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:44637
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=36501
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:29:08.587213 13852 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:29:08.591403 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap starting.
I20260810 19:29:08.611732 13852 log.cc:824] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:08.627821 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:29:08.628338 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d: Bootstrap complete.
I20260810 19:29:08.637918 13852 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:08.638764 13852 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Initialized, Role: FOLLOWER
I20260810 19:29:08.639415 13852 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:08.639621 13852 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:08.639745 13852 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:08.639904 13852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:29:08.643595 13852 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:08.644045 13852 leader_election.cc:302] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb941d2fd84748abb1234671a3ffc15d; no voters:
I20260810 19:29:08.644688 13852 leader_election.cc:288] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:29:08.644847 13859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:29:08.645828 13859 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [term 3 LEADER]: Becoming Leader. State: Replica: fb941d2fd84748abb1234671a3ffc15d, State: Running, Role: LEADER
I20260810 19:29:08.646355 13859 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } }
I20260810 19:29:08.646844 13852 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:08.648960 13863 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb941d2fd84748abb1234671a3ffc15d. Latest consensus state: current_term: 3 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:08.649096 13861 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb941d2fd84748abb1234671a3ffc15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb941d2fd84748abb1234671a3ffc15d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 39887 } } }
I20260810 19:29:08.649407 13863 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:08.649446 13861 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb941d2fd84748abb1234671a3ffc15d [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:08.650285 13871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:08.668331 13871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:08.670061 13871 catalog_manager.cc:1259] Loaded cluster ID: 43c88868dcb34424a23137119934dd8b
I20260810 19:29:08.670259 13871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 19:29:08.672360 13871 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe0932a29ad google::LogMessage::Fail() at ??:0
@ 0x7fe0932a588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe0932a24de google::LogMessage::Flush() at ??:0
@ 0x7fe0932a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe0a8cd5512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe0a8cd2976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe0a8d5a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe0a8d5a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe095eee345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe095ef191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe095ef1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe095ecceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe0991a66db start_thread at ??:0
@ 0x7fe09213c61f clone at ??:0
W20260810 19:29:08.948451 13857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:08.948746 13857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:08.948868 13857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:08.956352 13857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:08.956477 13857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:08.956640 13857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:29:08.967219 13857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:44637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=36501
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:08.968520 13857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:08.969873 13857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:08.982817 13923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:08.982849 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:08.983428 13929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:08.983861 13857 server_base.cc:1034] running on GCE node
I20260810 19:29:08.984424 13857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:08.985339 13857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:08.986541 13857 hybrid_clock.cc:648] HybridClock initialized: now 1786390148986494 us; error 41 us; skew 500 ppm
I20260810 19:29:08.986824 13857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:08 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390148, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:08.989955 13857 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:29:08.991307 13857 webserver.cc:460] Webserver started at http://127.19.68.1:36501/ using document root <none> and password file <none>
I20260810 19:29:08.991940 13857 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:08.992057 13857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:08.997063 13857 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:29:08.999886 13941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.001135 13857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:09.001308 13857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "94b10999eea8410fa5d5f7d9cdfbd485"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "cbb79cda6b4ede25e59f8d655453747b"
server_key_iv: "d5171aac4033e014adff56c75ca6d748"
server_key_version: "encryptionkey@0"
I20260810 19:29:09.002197 13857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:09.024986 13857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:09.028808 13857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:09.029072 13857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:09.030072 13857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:09.031774 13954 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 19:29:09.037022 13954 ts_tablet_manager.cc:1852] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 19:29:09.038096 13954 ts_tablet_manager.cc:1905] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:09.045634 13954 ts_tablet_manager.cc:1918] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:09.045827 13954 log.cc:1196] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/339862c5a1e84addb7125c44e3dfd1c6
I20260810 19:29:09.046314 13954 ts_tablet_manager.cc:1939] T 339862c5a1e84addb7125c44e3dfd1c6 P 94b10999eea8410fa5d5f7d9cdfbd485: Deleting consensus metadata
I20260810 19:29:09.046689 13857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 19:29:09.046823 13857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260810 19:29:09.046950 13857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:09.047024 13857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.078723 13857 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:44637
I20260810 19:29:09.078778 14128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:44637 every 8 connection(s)
I20260810 19:29:09.080050 13857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:29:09.083606 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 13857
I20260810 19:29:09.084424 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:37611
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=33443
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:29:09.085484 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.083056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:56837)
0810 19:29:09.083968 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.084262 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":340,"threads_started":1}
W20260810 19:29:09.086211 14130 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39887 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:39887: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
I20260810 19:29:09.087165 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.086781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:54649)
0810 19:29:09.086953 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.087039 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:29:09.088124 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.087716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:45811)
0810 19:29:09.087874 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.087971 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 19:29:09.088369 14130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:29:09.417440 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:09.417704 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:09.417826 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:09.424856 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:09.424955 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:09.425119 14137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:29:09.434675 14137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:37611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=33443
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:09.435989 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:09.437458 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:09.450354 14148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:09.450558 14146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:09.450414 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:09.451485 14137 server_base.cc:1034] running on GCE node
I20260810 19:29:09.452083 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:09.452975 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:09.454172 14137 hybrid_clock.cc:648] HybridClock initialized: now 1786390149454133 us; error 50 us; skew 500 ppm
I20260810 19:29:09.454464 14137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:09 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390149, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:09.457479 14137 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:29:09.458868 14137 webserver.cc:460] Webserver started at http://127.19.68.2:33443/ using document root <none> and password file <none>
I20260810 19:29:09.459512 14137 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:09.459630 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:09.464838 14137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:09.467613 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.469065 14137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:29:09.469302 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "5a8414bae776432db095180a414299a6"
format_stamp: "Formatted at 2026-08-10 19:29:03 on dist-test-slave-2j7r"
server_key: "1a16f31320ba476121a6ccb721491acd"
server_key_iv: "ffb42da7f83bc62ab255c7bfc387fe6d"
server_key_version: "encryptionkey@0"
I20260810 19:29:09.470322 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:09.502017 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:09.506379 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:09.506641 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:09.507689 14137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:09.509097 14137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:09.509204 14137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.509315 14137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:09.509387 14137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.540462 14137 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:37611
I20260810 19:29:09.540517 14340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:37611 every 8 connection(s)
I20260810 19:29:09.541924 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:29:09.544343 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 14137
I20260810 19:29:09.544950 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:42695
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:39887
--webserver_port=45829
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:29:09.546746 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.544754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:54603)
0810 19:29:09.545586 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.545840 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":345,"threads_started":1}
W20260810 19:29:09.547462 14341 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39887 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:39887: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
I20260810 19:29:09.548250 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.547930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:45043)
0810 19:29:09.548078 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.548144 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:29:09.549065 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.548773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:54545)
0810 19:29:09.548887 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.548961 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 19:29:09.549362 14341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:29:09.872454 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:09.872735 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:09.872869 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:09.880158 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:09.880275 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:09.880430 14347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:29:09.890368 14347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:42695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=45829
--tserver_master_addrs=127.19.68.62:39887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:09.891659 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:09.892951 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:09.905511 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:09.905455 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:09.905843 14361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:09.906461 14347 server_base.cc:1034] running on GCE node
I20260810 19:29:09.907068 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:09.907982 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:09.909183 14347 hybrid_clock.cc:648] HybridClock initialized: now 1786390149909132 us; error 41 us; skew 500 ppm
I20260810 19:29:09.909494 14347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:09 dist-test-slave-2j7r krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390149, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:09.912643 14347 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:29:09.914044 14347 webserver.cc:460] Webserver started at http://127.19.68.3:45829/ using document root <none> and password file <none>
I20260810 19:29:09.914688 14347 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:09.914806 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:09.919862 14347 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:29:09.922509 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.923727 14347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:29:09.923877 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "12b67a7443a84aef8a37d298882bbbf1"
format_stamp: "Formatted at 2026-08-10 19:29:04 on dist-test-slave-2j7r"
server_key: "f55fa7a9bf07462e4cff70942f36dec1"
server_key_iv: "b55a7942d15e1b62128b4d1a03d59861"
server_key_version: "encryptionkey@0"
I20260810 19:29:09.924844 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:09.940740 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:09.944777 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:09.945029 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:09.945986 14347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:09.947254 14347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:09.947358 14347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.947467 14347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:09.947546 14347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:09.978979 14347 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:42695
I20260810 19:29:09.979089 14558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:42695 every 8 connection(s)
I20260810 19:29:09.980357 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:29:09.982484 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 14347
I20260810 19:29:09.985451 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.983398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:34127)
0810 19:29:09.984298 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.984520 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
W20260810 19:29:09.986075 14563 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:39887 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:39887: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
I20260810 19:29:09.986812 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.986507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:42357)
0810 19:29:09.986639 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.986705 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:29:09.987550 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:09.987256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:55113)
0810 19:29:09.987370 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:09.987443 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 19:29:09.987767 14563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:29:10.089689 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:10.088867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:41207)
0810 19:29:10.089360 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:10.089445 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
I20260810 19:29:10.550777 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:10.549945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:44329)
0810 19:29:10.550433 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:10.550532 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":235,"threads_started":1}
I20260810 19:29:10.989107 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:10.988311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:33257)
0810 19:29:10.988740 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:10.988848 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":199,"threads_started":1}
I20260810 19:29:11.091280 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:11.090565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:46125)
0810 19:29:11.091038 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:11.091121 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260810 19:29:11.552304 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:11.551601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:49159)
0810 19:29:11.552053 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:11.552131 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":214,"threads_started":1}
I20260810 19:29:11.990572 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:11.989889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:46667)
0810 19:29:11.990320 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:11.990410 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260810 19:29:12.092867 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:12.092147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:60245)
0810 19:29:12.092603 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:12.092690 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":214,"threads_started":1}
I20260810 19:29:12.553984 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:12.553188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:36463)
0810 19:29:12.553704 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:12.553793 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260810 19:29:12.991977 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:12.991287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:56821)
0810 19:29:12.991725 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:12.991808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260810 19:29:13.094471 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:13.093743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:60609)
0810 19:29:13.094189 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:13.094268 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260810 19:29:13.555560 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:13.554837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:48623)
0810 19:29:13.555284 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:13.555364 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260810 19:29:13.993749 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:13.992891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:39907)
0810 19:29:13.993420 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:13.993513 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
I20260810 19:29:14.096112 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:14.095392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:60661)
0810 19:29:14.095845 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:14.095942 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260810 19:29:14.557286 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:14.556497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:42187)
0810 19:29:14.556977 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:14.557060 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260810 19:29:14.995342 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:14.994601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:51787)
0810 19:29:14.995088 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:14.995168 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":250,"threads_started":1}
I20260810 19:29:15.097745 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:15.096959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:37507)
0810 19:29:15.097501 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:15.097581 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":320,"threads_started":1}
I20260810 19:29:15.558789 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:15.558082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:52279)
0810 19:29:15.558537 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:15.558624 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260810 19:29:15.996852 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:15.996115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:52439)
0810 19:29:15.996582 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:15.996675 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":225,"threads_started":1}
I20260810 19:29:16.099479 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:16.098642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:37863)
0810 19:29:16.099163 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:16.099247 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":247,"threads_started":1}
I20260810 19:29:16.560374 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:16.559591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:58271)
0810 19:29:16.560081 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:16.560164 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":221,"threads_started":1}
I20260810 19:29:16.998660 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:16.997738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:49593)
0810 19:29:16.998308 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:16.998386 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260810 19:29:17.101102 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:17.100384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:52287)
0810 19:29:17.100814 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:17.100910 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260810 19:29:17.562033 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:17.561307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:49855)
0810 19:29:17.561773 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:17.561853 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":244,"threads_started":1}
I20260810 19:29:18.000355 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:17.999656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:33309)
0810 19:29:18.000097 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:18.000179 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260810 19:29:18.102707 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:18.101972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:55555)
0810 19:29:18.102432 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:18.102534 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":214,"threads_started":1}
I20260810 19:29:18.563755 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:18.562933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:46517)
0810 19:29:18.563454 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:18.563574 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":242,"threads_started":1}
I20260810 19:29:19.002007 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:19.001250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:35509)
0810 19:29:19.001736 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:19.001821 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260810 19:29:19.104395 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:19.103665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:48461)
0810 19:29:19.104104 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:19.104188 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260810 19:29:19.565507 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:19.564665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:44175)
0810 19:29:19.565241 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:19.565332 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
I20260810 19:29:20.003680 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:20.002930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:59575)
0810 19:29:20.003412 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:20.003507 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":280,"threads_started":1}
I20260810 19:29:20.106132 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:20.105296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:58845)
0810 19:29:20.105829 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:20.105921 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
I20260810 19:29:20.567413 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:20.566465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:35303)
0810 19:29:20.567094 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:20.567197 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":388,"threads_started":1}
I20260810 19:29:21.005515 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:21.004667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:55121)
0810 19:29:21.005203 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:21.005322 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":254,"threads_started":1}
I20260810 19:29:21.107764 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:21.107059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:33891)
0810 19:29:21.107507 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:21.107599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260810 19:29:21.569380 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:21.568606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:41187)
0810 19:29:21.569076 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:21.569210 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260810 19:29:22.007167 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:22.006444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:50291)
0810 19:29:22.006910 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:22.007002 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260810 19:29:22.109371 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:22.108615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:58595)
0810 19:29:22.109040 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:22.109176 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":204,"threads_started":1}
I20260810 19:29:22.571153 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:22.570333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:54289)
0810 19:29:22.570845 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:22.570940 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":246,"threads_started":1}
I20260810 19:29:23.008927 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:23.008041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:44509)
0810 19:29:23.008580 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:23.008723 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
I20260810 19:29:23.110987 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:23.110243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:36933)
0810 19:29:23.110712 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:23.110795 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
I20260810 19:29:23.572880 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:23.572070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:38521)
0810 19:29:23.572598 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:23.572683 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260810 19:29:24.010540 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:24.009809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:36685)
0810 19:29:24.010269 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:24.010353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260810 19:29:24.112788 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:24.112040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.1:44523)
0810 19:29:24.112484 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:24.112596 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":213,"threads_started":1}
I20260810 19:29:24.574926 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:24.574039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.2:44383)
0810 19:29:24.574586 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:24.574717 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
I20260810 19:29:24.989111 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 13857
I20260810 19:29:25.012169 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:24.990191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.19.68.1:44637)
0810 19:29:24.990825 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:24.990844 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:24.990898 (+ 54us) server_negotiation.cc:407] Connection header received
0810 19:29:24.992953 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:24.992962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:24.993448 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:24.993731 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:24.995098 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:24.996340 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:24.997558 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:24.998217 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:24.998365 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:25.006097 (+ 7732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:25.006105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:25.006121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:25.006159 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:25.008969 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.009696 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.009703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.009709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.009891 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.010203 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.010206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.010208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.010752 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:25.011038 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:25.011483 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 19:29:25.011877 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260810 19:29:25.012449 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:25.011678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:39887 (local address 127.19.68.3:53863)
0810 19:29:25.012142 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:25.012233 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260810 19:29:25.013568 14123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:25.097743 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 13857
I20260810 19:29:25.123641 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 14137
I20260810 19:29:25.141253 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:25.124667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.19.68.2:37611)
0810 19:29:25.125246 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:25.125262 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:25.125301 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:29:25.126876 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:25.126886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:25.127311 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:25.127653 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:25.129375 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:25.130639 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:25.131923 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:25.132584 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:25.132714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:25.135344 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:25.135351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:25.135365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:25.135402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:25.138179 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.138835 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.138841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.138847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.139037 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.139447 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.139450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.139452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.139975 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:25.140195 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:25.140578 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 19:29:25.140929 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
I20260810 19:29:25.142571 14334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:25.225751 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 14137
I20260810 19:29:25.251883 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 14347
I20260810 19:29:25.269769 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:25.253057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.19.68.3:42695)
0810 19:29:25.253765 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:25.253780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:25.253822 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:29:25.255459 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:25.255469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:25.255887 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:25.256166 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:25.257748 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:25.258975 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:25.260260 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:25.260902 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:25.261041 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:25.263809 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:25.263816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:25.263829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:25.263865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:25.266763 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.267474 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.267480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.267485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.267641 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:25.267995 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:25.267998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:25.268000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:25.268497 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:25.268718 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:25.269081 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 19:29:25.269477 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":357,"threads_started":1}
I20260810 19:29:25.271289 14551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:25.352468 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 14347
2026-08-10T19:29:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24828 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:29:25 dist-test-slave-2j7r krb5kdc[14662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:29:25 dist-test-slave-2j7r krb5kdc[14662](info): set up 2 sockets
Aug 10 19:29:25 dist-test-slave-2j7r krb5kdc[14662](info): commencing operation
krb5kdc: starting...
W20260810 19:29:26.620558 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:29:26 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:29:26.914081 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:27.262601 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:27.262893 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:27.263033 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:27.270166 14682 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:29:27.270270 14682 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:29:27.270344 14682 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:29:27.270412 14682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:27.270488 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:27.270581 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:27.270644 14682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:27.270720 14682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:27.280253 14682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:27.281620 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:27.282948 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:27.294667 14693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:27.294705 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:27.294754 14691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:27.295915 14682 server_base.cc:1034] running on GCE node
I20260810 19:29:27.296631 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:27.298118 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:27.299350 14682 hybrid_clock.cc:648] HybridClock initialized: now 1786390167299288 us; error 37 us; skew 500 ppm
I20260810 19:29:27.299646 14682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:27 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:27.302708 14682 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:27.304108 14682 webserver.cc:460] Webserver started at http://127.19.68.62:39489/ using document root <none> and password file <none>
I20260810 19:29:27.304740 14682 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:27.304826 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:27.305135 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:27.308089 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "2886c323f59649058c4b1c3ae9c13ed4"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "10f93c2ce6a200064b02482a76fe75dc"
server_key_iv: "1300cf7d87c419fcb8b367e850627968"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.308796 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "2886c323f59649058c4b1c3ae9c13ed4"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "10f93c2ce6a200064b02482a76fe75dc"
server_key_iv: "1300cf7d87c419fcb8b367e850627968"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.313817 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:29:27.317131 14705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:27.318524 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:29:27.318715 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2886c323f59649058c4b1c3ae9c13ed4"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "10f93c2ce6a200064b02482a76fe75dc"
server_key_iv: "1300cf7d87c419fcb8b367e850627968"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.318971 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:27.352396 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:27.356870 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:27.357167 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:27.375214 14682 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:36725
I20260810 19:29:27.375273 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:36725 every 8 connection(s)
I20260810 19:29:27.376602 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:27.380271 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:27.382592 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 14682
I20260810 19:29:27.382822 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:29:27.383680 19728 external_mini_cluster.cc:1428] Setting key 3ad31606cc882a2c612862005cd45ff6
I20260810 19:29:27.392691 14785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap starting.
I20260810 19:29:27.396332 14785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:27.397640 14785 log.cc:824] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:29:27 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:27.400916 14785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: No bootstrap required, opened a new log
I20260810 19:29:27.407868 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:27.386075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.19.68.62:36725)
0810 19:29:27.386900 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:27.386911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:27.386964 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:29:27.388755 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:27.388777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:27.389262 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:27.389604 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:27.391369 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:27.392836 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:27.394739 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:27.395448 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:27.395586 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:27.400670 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:27.400678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:27.400691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:27.400727 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:27.403810 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:27.404636 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:27.404642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:27.404647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:27.404814 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:27.405221 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:27.405225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:27.405227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:27.405741 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:27.406005 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:27.406482 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 19:29:27.406830 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":367,"threads_started":1}
I20260810 19:29:27.409432 14785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:27.409770 14785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:27.409878 14785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Initialized, Role: FOLLOWER
I20260810 19:29:27.410439 14785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:27.410686 14785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:27.410818 14785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:27.410970 14785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:27.413046 14785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:27.413530 14785 leader_election.cc:302] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886c323f59649058c4b1c3ae9c13ed4; no voters:
I20260810 19:29:27.414100 14785 leader_election.cc:288] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:29:27.414242 14790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:27.414939 14790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Running, Role: LEADER
I20260810 19:29:27.415470 14790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:27.416080 14785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:27.418285 14793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886c323f59649058c4b1c3ae9c13ed4. Latest consensus state: current_term: 1 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:29:27.418706 14793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:27.419261 14792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:29:27.419551 14792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:27.422416 14800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:27.427795 14800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:27.436667 14800 catalog_manager.cc:1347] Generated new cluster ID: 74e218db8f4349589f6d788e01aeab77
I20260810 19:29:27.436764 14800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:29:27.449193 14800 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:29:27.451023 14800 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:29:27.473361 14800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Generated new TSK 0
I20260810 19:29:27.474395 14800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:29:27.546561 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:27.881561 14820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:27.881845 14820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:27.881968 14820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:27.889264 14820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:27.889385 14820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:27.889511 14820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:29:27.899402 14820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:27.900629 14820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:27.901988 14820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:27.915467 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:27.915557 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:27.915586 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:27.921444 14820 server_base.cc:1034] running on GCE node
I20260810 19:29:27.922004 14820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:27.922976 14820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:27.924170 14820 hybrid_clock.cc:648] HybridClock initialized: now 1786390167924112 us; error 46 us; skew 500 ppm
I20260810 19:29:27.924474 14820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:27 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:27.927510 14820 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:29:27.928979 14820 webserver.cc:460] Webserver started at http://127.19.68.1:40517/ using document root <none> and password file <none>
I20260810 19:29:27.929689 14820 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:27.929806 14820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:27.930101 14820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:27.933358 14820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "4e9299fca4b8491f9be4a1a7252a772c"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "2c1eb5354697282a6235603f4546b02a"
server_key_iv: "8e3ce12ec75af5644d3af12f7d8ddb98"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.934255 14820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "4e9299fca4b8491f9be4a1a7252a772c"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "2c1eb5354697282a6235603f4546b02a"
server_key_iv: "8e3ce12ec75af5644d3af12f7d8ddb98"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.940177 14820 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 19:29:27.943691 14846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:27.945027 14820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:29:27.945206 14820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "4e9299fca4b8491f9be4a1a7252a772c"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "2c1eb5354697282a6235603f4546b02a"
server_key_iv: "8e3ce12ec75af5644d3af12f7d8ddb98"
server_key_version: "encryptionkey@0"
I20260810 19:29:27.945410 14820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:27.973443 14820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:27.977536 14820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:27.977841 14820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:27.979707 14820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:27.981019 14820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:27.981115 14820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:27.981236 14820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:27.981290 14820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.011757 14820 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:33981
I20260810 19:29:28.011827 15042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:33981 every 8 connection(s)
I20260810 19:29:28.012976 14820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:29:28.015067 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 14820
I20260810 19:29:28.015259 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:29:28.016068 19728 external_mini_cluster.cc:1428] Setting key 06349f1f6cbd0200481f4a156f6c9a00
Aug 10 19:29:28 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:28.033130 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:28.015543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58615 (local address 127.19.68.62:36725)
0810 19:29:28.016071 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:28.016078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:28.017132 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 19:29:28.019072 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:28.019077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:28.019151 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:28.019275 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:28.020639 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.021604 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.023374 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.023916 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.024084 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:28.027587 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:28.027592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:28.027595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:28.027628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:28.030044 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:28.030828 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:28.030833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:28.030836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:28.031006 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:28.031393 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:28.031398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:28.031401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:28.031716 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:28.031873 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:28.032515 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 19:29:28.032729 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260810 19:29:28.033917 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:28.016121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:58615)
0810 19:29:28.016933 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:28.016990 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:28.018811 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:28.019433 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:28.019440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:28.019933 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:28.020470 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:28.020487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.021764 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.021768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:28.023242 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:28.023261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.024118 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.024123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:28.024309 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:28.024915 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:28.024936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:28.027380 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:28.030173 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:28.030181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:28.030194 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:28.030718 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:28.031108 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:28.031112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:28.031114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:28.031301 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:28.031822 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:28.031828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:28.032111 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 19:29:28.032389 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 19:29:28.032757 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":330,"threads_started":1}
I20260810 19:29:28.035451 15044 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:28.035754 15044 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:28.036573 15044 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:28.038952 14730 ts_manager.cc:194] Registered new tserver with Master: 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1:33981)
I20260810 19:29:28.040457 14730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:58615
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:29:28.081902 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:28.406193 15059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:28.406477 15059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:28.406620 15059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:28.413683 15059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:28.413798 15059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:28.413952 15059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:29:28.423785 15059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:28.425107 15059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:28.426440 15059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:28.438685 15074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:28.438755 15071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:28.438726 15072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:28.440038 15059 server_base.cc:1034] running on GCE node
I20260810 19:29:28.440594 15059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:28.441488 15059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:28.442667 15059 hybrid_clock.cc:648] HybridClock initialized: now 1786390168442612 us; error 48 us; skew 500 ppm
I20260810 19:29:28.442951 15059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:28 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:28.446031 15059 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:29:28.447474 15059 webserver.cc:460] Webserver started at http://127.19.68.2:39203/ using document root <none> and password file <none>
I20260810 19:29:28.448107 15059 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:28.448225 15059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:28.448513 15059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:28.451436 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "3e423b694aec401583ab098207086c8d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "d2ac411e2f507b62814f7690951ccf51"
server_key_iv: "be817cdc86b867f365e6984548f5568b"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.452144 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "3e423b694aec401583ab098207086c8d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "d2ac411e2f507b62814f7690951ccf51"
server_key_iv: "be817cdc86b867f365e6984548f5568b"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.457113 15059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 19:29:28.460464 15084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.461804 15059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:29:28.461956 15059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "3e423b694aec401583ab098207086c8d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "d2ac411e2f507b62814f7690951ccf51"
server_key_iv: "be817cdc86b867f365e6984548f5568b"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.462138 15059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:28.476341 15059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:28.480765 15059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:28.481057 15059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:28.482134 15059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:28.483544 15059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:28.483639 15059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.483772 15059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:28.483845 15059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.517005 15059 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:41029
I20260810 19:29:28.517120 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:41029 every 8 connection(s)
I20260810 19:29:28.518679 15059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:29:28.520354 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 15059
I20260810 19:29:28.520539 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:29:28.521063 19728 external_mini_cluster.cc:1428] Setting key f8866b34057a5148ab655cbabf36e57b
Aug 10 19:29:28 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:28.540059 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:28.521471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:48973 (local address 127.19.68.62:36725)
0810 19:29:28.521716 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:28.521722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:28.523175 (+ 1453us) server_negotiation.cc:407] Connection header received
0810 19:29:28.525163 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:28.525169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:28.525250 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:28.525348 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:28.526896 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.527797 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.529495 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.530057 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.530175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:28.534234 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:28.534239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:28.534242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:28.534277 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:28.536957 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:28.537680 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:28.537685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:28.537688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:28.537837 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:28.538229 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:28.538234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:28.538238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:28.538565 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:28.538715 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:28.539372 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 19:29:28.539624 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 19:29:28.540652 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:28.522050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:48973)
0810 19:29:28.523007 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:28.523056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:28.524869 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:28.525558 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:28.525566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:28.526134 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:28.526714 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:28.526735 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.527968 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.527972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:28.529312 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:28.529328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:28.530284 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:28.530292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:28.530553 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:28.531249 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:28.531279 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:28.533991 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:28.537119 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:28.537128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:28.537145 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:28.537564 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:28.537934 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:28.537938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:28.537940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:28.538123 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:28.538674 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:28.538680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:28.539007 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 19:29:28.539214 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 19:29:28.539596 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":423,"threads_started":1}
I20260810 19:29:28.542001 15276 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:28.542356 15276 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:28.543205 15276 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:28.544853 14730 ts_manager.cc:194] Registered new tserver with Master: 3e423b694aec401583ab098207086c8d (127.19.68.2:41029)
I20260810 19:29:28.545706 14730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:48973
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:29:28.588939 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:28.915920 15282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:28.916175 15282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:28.916306 15282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:28.923455 15282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:28.923550 15282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:28.923694 15282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:29:28.933226 15282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:28.934466 15282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:28.935722 15282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:28.948495 15294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:28.948604 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:28.948566 15291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:28.949213 15282 server_base.cc:1034] running on GCE node
I20260810 19:29:28.950045 15282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:28.950982 15282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:28.952193 15282 hybrid_clock.cc:648] HybridClock initialized: now 1786390168952133 us; error 43 us; skew 500 ppm
I20260810 19:29:28.952502 15282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:28 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:28.955708 15282 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:29:28.957048 15282 webserver.cc:460] Webserver started at http://127.19.68.3:44297/ using document root <none> and password file <none>
I20260810 19:29:28.957742 15282 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:28.957856 15282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:28.958148 15282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:29:28.961172 15282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "c57bcbf6a3a2415750f0e6bcb68b90ff"
server_key_iv: "857015dcf636a49969b768d389a2b732"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.961953 15282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "c57bcbf6a3a2415750f0e6bcb68b90ff"
server_key_iv: "857015dcf636a49969b768d389a2b732"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.966957 15282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 19:29:28.970207 15302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.971402 15282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 19:29:28.971544 15282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "c57bcbf6a3a2415750f0e6bcb68b90ff"
server_key_iv: "857015dcf636a49969b768d389a2b732"
server_key_version: "encryptionkey@0"
I20260810 19:29:28.971709 15282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:28.991075 15282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:28.994758 15282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:28.995035 15282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:28.995956 15282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:28.997241 15282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:28.997341 15282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:28.997457 15282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:28.997506 15282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:29.028873 15282 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:38007
I20260810 19:29:29.028928 15497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:38007 every 8 connection(s)
I20260810 19:29:29.030133 15282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:29:29.036018 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 15282
I20260810 19:29:29.036218 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:29:29.036852 19728 external_mini_cluster.cc:1428] Setting key ef51e1dc89886b7d7adacc969ca1bad5
Aug 10 19:29:29 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:29.043857 15044 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:29.050388 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.032550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:51943 (local address 127.19.68.62:36725)
0810 19:29:29.032771 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.032777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.033945 (+ 1168us) server_negotiation.cc:407] Connection header received
0810 19:29:29.035807 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.035811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.035882 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.035974 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:29.037968 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.038822 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.040555 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.040971 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.041064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.044404 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:29.044409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:29.044412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:29.044446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:29.047275 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.047987 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.047991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.047994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.048129 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.048576 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.048581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.048584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.048849 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:29.048985 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.049750 (+ 765us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.049973 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:29:29.051539 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.032971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:51943)
0810 19:29:29.033777 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.033826 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.035490 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.036258 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.036270 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.037015 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:29.037778 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.037806 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.038978 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.038983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.040429 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.040442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.041202 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.041208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.041440 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.042011 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:29.042039 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:29.044199 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:29.047406 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:29.047412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:29.047425 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:29.047858 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:29.048248 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:29.048254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:29.048258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:29.048454 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:29.048986 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:29.048993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:29.049378 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.049614 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.050161 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":311,"threads_started":1}
I20260810 19:29:29.052983 15502 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:29.053268 15502 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:29.053946 15502 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:29.055263 14730 ts_manager.cc:194] Registered new tserver with Master: 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007)
I20260810 19:29:29.056068 14730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:51943
I20260810 19:29:29.064493 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:29:29.080927 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.069041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.19.68.62:36725)
0810 19:29:29.069276 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.069283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.069661 (+ 378us) server_negotiation.cc:407] Connection header received
0810 19:29:29.069790 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.069794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.069851 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.069930 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:29.071371 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.072087 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.073498 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.073894 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.073982 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.076274 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:29.076279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:29.076282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:29.076312 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:29.078687 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.079284 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.079288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.079290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.079415 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.079698 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.079701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.079702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.079986 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:29.080166 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.080503 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.080660 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 19:29:29.084743 14730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:29:29.091508 14730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:29:29.120146 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.102962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.19.68.2:41029)
0810 19:29:29.109685 (+ 6723us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.109694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.110891 (+ 1197us) server_negotiation.cc:407] Connection header received
0810 19:29:29.111010 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.111019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.111744 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.111948 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:29.112973 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.114551 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.117234 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.119021 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.119153 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.119373 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.119678 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.119844 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6564,"thread_start_us":6462,"threads_started":1}
I20260810 19:29:29.120210 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.102820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:41029 (local address 127.0.0.1:51758)
0810 19:29:29.110416 (+ 7596us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.110464 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.110680 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.112257 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.112282 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.112321 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:29.112765 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.112781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.114686 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.114691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.117023 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.117039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.119238 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.119244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.119435 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.119490 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.119609 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.119805 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7313,"thread_start_us":7174,"threads_started":1}
I20260810 19:29:29.121037 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.103618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.19.68.1:33981)
0810 19:29:29.113192 (+ 9574us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.113201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.113228 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:29:29.113320 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.113327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.113756 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.113953 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:29.114751 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.115937 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.118160 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.119062 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.119181 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.119416 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.120548 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.120709 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9361,"thread_start_us":9186,"threads_started":1}
I20260810 19:29:29.121198 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.102957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:58234)
0810 19:29:29.112410 (+ 9453us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.112429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.112559 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.114189 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.114194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.114217 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:29.114583 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.114593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.116059 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.116062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.118007 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.118019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.119540 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.119546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.119722 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.119754 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.119840 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.119967 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9319,"thread_start_us":10236,"threads_started":1}
I20260810 19:29:29.121604 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.104329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38007 (local address 127.0.0.1:42976)
0810 19:29:29.113462 (+ 9133us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.113478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.113599 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.114482 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.114486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.114515 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:29.115029 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.115043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.117025 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.117030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.119131 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.119145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.120867 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.120873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.121029 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.121061 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.121171 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.121294 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9007,"thread_start_us":7816,"threads_started":1}
I20260810 19:29:29.121969 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.104941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.19.68.3:38007)
0810 19:29:29.111996 (+ 7055us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.112004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.113717 (+ 1713us) server_negotiation.cc:407] Connection header received
0810 19:29:29.113828 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.113835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.114047 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.114205 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:29.115256 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.116837 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.119316 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.120664 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.120788 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.121016 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.121415 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.121617 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6876,"thread_start_us":6718,"threads_started":1}
I20260810 19:29:29.124363 15167 tablet_service.cc:1467] Processing CreateTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 (DEFAULT_TABLE table=test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:29.125167 15382 tablet_service.cc:1467] Processing CreateTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 (DEFAULT_TABLE table=test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:29.125000 14937 tablet_service.cc:1467] Processing CreateTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 (DEFAULT_TABLE table=test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:29.125922 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf968e78dfc4cb1a80fdc8d5e533258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:29.126334 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf968e78dfc4cb1a80fdc8d5e533258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:29.126444 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf968e78dfc4cb1a80fdc8d5e533258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:29.138087 15528 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Bootstrap starting.
I20260810 19:29:29.140146 15530 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Bootstrap starting.
I20260810 19:29:29.141811 15528 tablet_bootstrap.cc:654] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:29.143247 15528 log.cc:824] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:29.143502 15530 tablet_bootstrap.cc:654] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:29.144534 15530 log.cc:824] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:29.146656 15528 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: No bootstrap required, opened a new log
I20260810 19:29:29.146937 15528 ts_tablet_manager.cc:1397] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 19:29:29.147523 15530 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: No bootstrap required, opened a new log
I20260810 19:29:29.147768 15530 ts_tablet_manager.cc:1397] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260810 19:29:29.149211 15533 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Bootstrap starting.
I20260810 19:29:29.152163 15533 tablet_bootstrap.cc:654] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:29.153065 15533 log.cc:824] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:29.155393 15533 tablet_bootstrap.cc:492] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: No bootstrap required, opened a new log
I20260810 19:29:29.155622 15533 ts_tablet_manager.cc:1397] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260810 19:29:29.157251 15528 raft_consensus.cc:348] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.157691 15528 raft_consensus.cc:374] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:29.157789 15528 raft_consensus.cc:729] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8cce502d574cf6bcdb79254ebbab9d, State: Initialized, Role: FOLLOWER
I20260810 19:29:29.157670 15530 raft_consensus.cc:348] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.158093 15530 raft_consensus.cc:374] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:29.158198 15530 raft_consensus.cc:729] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e423b694aec401583ab098207086c8d, State: Initialized, Role: FOLLOWER
I20260810 19:29:29.158483 15528 consensus_queue.cc:260] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.158738 15530 consensus_queue.cc:260] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.159642 15502 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:29.162120 15528 ts_tablet_manager.cc:1428] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 19:29:29.163007 15276 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:29.163501 15533 raft_consensus.cc:348] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.163898 15533 raft_consensus.cc:374] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:29.163982 15530 ts_tablet_manager.cc:1428] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260810 19:29:29.163995 15533 raft_consensus.cc:729] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9299fca4b8491f9be4a1a7252a772c, State: Initialized, Role: FOLLOWER
I20260810 19:29:29.164601 15533 consensus_queue.cc:260] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.166373 15533 ts_tablet_manager.cc:1428] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
W20260810 19:29:29.265365 15045 tablet.cc:2369] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:29:29.270669 15277 tablet.cc:2369] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:29:29.281714 15504 tablet.cc:2369] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:29:29.306591 15541 raft_consensus.cc:482] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:29:29.306882 15541 raft_consensus.cc:504] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.308681 15541 leader_election.cc:288] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e423b694aec401583ab098207086c8d (127.19.68.2:41029), 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007)
I20260810 19:29:29.317781 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.309216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:52795 (local address 127.19.68.2:41029)
0810 19:29:29.309423 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.309429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.310155 (+ 726us) server_negotiation.cc:407] Connection header received
0810 19:29:29.310378 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.310383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.310453 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.310548 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:29.311304 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.312444 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.315267 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.316541 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.316687 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.316776 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.317397 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.317525 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:29:29.317747 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.309228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:45545 (local address 127.19.68.3:38007)
0810 19:29:29.309439 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.309444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.310154 (+ 710us) server_negotiation.cc:407] Connection header received
0810 19:29:29.310357 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.310363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.310447 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.310548 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:29.311311 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.312412 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.315267 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.316541 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.316678 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.316780 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.317352 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.317487 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:29:29.317962 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.309182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38007 (local address 127.19.68.1:45545)
0810 19:29:29.310056 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.310072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.310199 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.310584 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.310589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.310645 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:29.311137 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.311147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.312557 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.312561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.315073 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.315090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.316763 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.316811 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.317101 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.317197 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.317306 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.317457 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":269,"spinlock_wait_cycles":31616,"thread_start_us":447,"threads_started":1}
I20260810 19:29:29.317931 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.309150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:41029 (local address 127.19.68.1:52795)
0810 19:29:29.310021 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:29.310044 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:29.310233 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:29.310584 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:29.310589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:29.310620 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:29.311107 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.311124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.312555 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.312561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.315073 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:29.315090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.316844 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.316850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:29.317102 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.317192 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 19:29:29.317306 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:29:29.317457 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":176,"spinlock_wait_cycles":23936,"thread_start_us":509,"threads_started":1}
I20260810 19:29:29.319058 15416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf968e78dfc4cb1a80fdc8d5e533258" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" is_pre_election: true
I20260810 19:29:29.319132 15194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf968e78dfc4cb1a80fdc8d5e533258" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e423b694aec401583ab098207086c8d" is_pre_election: true
I20260810 19:29:29.319553 15416 raft_consensus.cc:2454] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 0.
I20260810 19:29:29.319590 15194 raft_consensus.cc:2454] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 0.
I20260810 19:29:29.320191 14847 leader_election.cc:302] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e423b694aec401583ab098207086c8d, 4d8cce502d574cf6bcdb79254ebbab9d, 4e9299fca4b8491f9be4a1a7252a772c; no voters:
I20260810 19:29:29.320571 15541 raft_consensus.cc:2781] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:29:29.320709 15541 raft_consensus.cc:482] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:29:29.320808 15541 raft_consensus.cc:3037] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:29.323067 15541 raft_consensus.cc:504] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.323721 15541 leader_election.cc:288] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 election: Requested vote from peers 3e423b694aec401583ab098207086c8d (127.19.68.2:41029), 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007)
I20260810 19:29:29.324127 15194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf968e78dfc4cb1a80fdc8d5e533258" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e423b694aec401583ab098207086c8d"
I20260810 19:29:29.324174 15416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf968e78dfc4cb1a80fdc8d5e533258" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
I20260810 19:29:29.324295 15194 raft_consensus.cc:3037] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:29.324363 15416 raft_consensus.cc:3037] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:29.326567 15194 raft_consensus.cc:2454] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 1.
I20260810 19:29:29.326591 15416 raft_consensus.cc:2454] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 1.
I20260810 19:29:29.326973 14849 leader_election.cc:302] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e423b694aec401583ab098207086c8d, 4e9299fca4b8491f9be4a1a7252a772c; no voters:
I20260810 19:29:29.327273 15541 raft_consensus.cc:2781] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:29.327672 15541 raft_consensus.cc:686] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 LEADER]: Becoming Leader. State: Replica: 4e9299fca4b8491f9be4a1a7252a772c, State: Running, Role: LEADER
I20260810 19:29:29.328112 15541 consensus_queue.cc:237] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } }
I20260810 19:29:29.332756 14730 catalog_manager.cc:5495] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4e9299fca4b8491f9be4a1a7252a772c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } health_report { overall_health: UNKNOWN } } }
I20260810 19:29:29.358048 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.353259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.19.68.1:33981)
0810 19:29:29.353444 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.353449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.353466 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:29:29.353631 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.353635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.353709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.353851 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:29.354147 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.354890 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.356218 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.356596 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.356685 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.357060 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:29.357605 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:29:29.357708 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.357774 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.357853 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:29:29.365641 15416 raft_consensus.cc:1261] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Refusing update from remote peer 4e9299fca4b8491f9be4a1a7252a772c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:29.365721 15194 raft_consensus.cc:1261] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Refusing update from remote peer 4e9299fca4b8491f9be4a1a7252a772c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:29.366608 15546 consensus_queue.cc:1035] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:29.366958 15541 consensus_queue.cc:1035] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:29.377826 15550 mvcc.cc:205] Tried to move back new op lower bound from 7317054133712244736 to 7317054133571440640. Current Snapshot: MvccSnapshot[applied={T|T < 7317054133712244736}]
I20260810 19:29:29.382251 15551 mvcc.cc:205] Tried to move back new op lower bound from 7317054133712244736 to 7317054133571440640. Current Snapshot: MvccSnapshot[applied={T|T < 7317054133712244736}]
I20260810 19:29:29.388530 15554 mvcc.cc:205] Tried to move back new op lower bound from 7317054133712244736 to 7317054133571440640. Current Snapshot: MvccSnapshot[applied={T|T < 7317054133712244736}]
I20260810 19:29:29.409217 14730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:29.409438 14730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:29.412930 14730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Sending DeleteTablet for 3 replicas of tablet caf968e78dfc4cb1a80fdc8d5e533258
I20260810 19:29:29.414105 15167 tablet_service.cc:1514] Processing DeleteTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:29 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:51758
I20260810 19:29:29.414359 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 14820
I20260810 19:29:29.414290 15382 tablet_service.cc:1514] Processing DeleteTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:29 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:42976
I20260810 19:29:29.414790 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet caf968e78dfc4cb1a80fdc8d5e533258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:29 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:58234
I20260810 19:29:29.438565 15569 tablet_replica.cc:330] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: stopping tablet replica
I20260810 19:29:29.439277 15569 raft_consensus.cc:2229] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:29:29.440104 15569 raft_consensus.cc:2258] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:29.440735 15564 tablet_replica.cc:330] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: stopping tablet replica
I20260810 19:29:29.442418 15564 raft_consensus.cc:2229] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:29.444761 15569 ts_tablet_manager.cc:1905] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:29.445381 15564 raft_consensus.cc:2258] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:29.447660 15564 ts_tablet_manager.cc:1905] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:29:29 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:29:29.459400 15569 ts_tablet_manager.cc:1918] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:29.460108 15565 tablet_replica.cc:330] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: stopping tablet replica
I20260810 19:29:29.460664 15565 raft_consensus.cc:2229] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:29.461369 15565 raft_consensus.cc:2258] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:29.462127 15569 log.cc:1196] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/caf968e78dfc4cb1a80fdc8d5e533258
I20260810 19:29:29.462749 15569 ts_tablet_manager.cc:1939] T caf968e78dfc4cb1a80fdc8d5e533258 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting consensus metadata
I20260810 19:29:29.463472 15564 ts_tablet_manager.cc:1918] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:29.463650 15564 log.cc:1196] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/caf968e78dfc4cb1a80fdc8d5e533258
I20260810 19:29:29.464170 15564 ts_tablet_manager.cc:1939] T caf968e78dfc4cb1a80fdc8d5e533258 P 3e423b694aec401583ab098207086c8d: Deleting consensus metadata
I20260810 19:29:29.464344 15565 ts_tablet_manager.cc:1905] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:29.465009 14710 catalog_manager.cc:4841] TS 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1:33981): tablet caf968e78dfc4cb1a80fdc8d5e533258 (table test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]) successfully deleted
I20260810 19:29:29.465866 14709 catalog_manager.cc:4841] TS 3e423b694aec401583ab098207086c8d (127.19.68.2:41029): tablet caf968e78dfc4cb1a80fdc8d5e533258 (table test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]) successfully deleted
I20260810 19:29:29.466988 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.415675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.19.68.1:33981)
0810 19:29:29.415899 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.415905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.441527 (+ 25622us) server_negotiation.cc:407] Connection header received
0810 19:29:29.441736 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.441743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.441846 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.441964 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:29.444222 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.445531 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.447648 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.448409 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.448512 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.454677 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:29.454693 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:29.454714 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:29.454764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:29.459293 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.460281 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.460290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.460293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.460507 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.460991 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.460996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.460999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.462574 (+ 1575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:29.462859 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.466344 (+ 3485us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.466603 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:29:29.467682 15036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:29.472184 15565 ts_tablet_manager.cc:1918] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:29.472318 15565 log.cc:1196] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/caf968e78dfc4cb1a80fdc8d5e533258
I20260810 19:29:29.472715 15565 ts_tablet_manager.cc:1939] T caf968e78dfc4cb1a80fdc8d5e533258 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting consensus metadata
I20260810 19:29:29.474268 14706 catalog_manager.cc:4841] TS 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007): tablet caf968e78dfc4cb1a80fdc8d5e533258 (table test-table [id=7f0df7b8cd8c4672861c7f9519ca020e]) successfully deleted
I20260810 19:29:29.580839 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 14820
I20260810 19:29:29.606436 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 15059
Aug 10 19:29:29 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:29:29.621317 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.607169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.19.68.2:41029)
0810 19:29:29.607352 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.607356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.607419 (+ 63us) server_negotiation.cc:407] Connection header received
0810 19:29:29.607582 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.607586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.607665 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.607746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:29.609054 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.609807 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.611262 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.611629 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.611714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.616049 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:29.616065 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:29.616080 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:29.616117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:29.618882 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.619487 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.619494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.619496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.619627 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.619923 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.619925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.619927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.620289 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:29.620520 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.620762 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.620971 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:29:29.621994 15262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:29.709856 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 15059
I20260810 19:29:29.736439 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 15282
Aug 10 19:29:29 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:29:29.751482 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:29.737368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.19.68.3:38007)
0810 19:29:29.737541 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:29.737545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:29.737620 (+ 75us) server_negotiation.cc:407] Connection header received
0810 19:29:29.737773 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:29.737778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:29.737847 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:29.737943 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:29.739357 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.740077 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.741573 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:29.741968 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:29.742060 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:29.746160 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:29.746176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:29.746192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:29.746225 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:29.748825 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.749643 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.749651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.749653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.749801 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:29.750129 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:29.750132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:29.750134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:29.750504 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:29.750729 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:29.751040 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:29:29.751200 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:29:29.752107 15488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:29.840024 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 15282
I20260810 19:29:29.864887 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 14682
I20260810 19:29:29.866153 14779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:29.975136 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 14682
I20260810 19:29:50.002348 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--webserver_interface=127.19.68.62
--webserver_port=39489
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:29:50.383919 15577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:50.384187 15577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:50.384297 15577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:50.391543 15577 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:29:50.391630 15577 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:29:50.391681 15577 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:29:50.391733 15577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:29:50.391790 15577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:50.391897 15577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:50.391968 15577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:29:50.392043 15577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:29:50.404495 15577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:50.406260 15577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:50.407946 15577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:50.434011 15584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:50.433926 15585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:50.433895 15587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:50.435081 15577 server_base.cc:1034] running on GCE node
I20260810 19:29:50.436154 15577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:50.438175 15577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:50.439472 15577 hybrid_clock.cc:648] HybridClock initialized: now 1786390190439392 us; error 50 us; skew 500 ppm
I20260810 19:29:50.440006 15577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:50 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390190, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:50.444200 15577 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:29:50.445865 15577 webserver.cc:460] Webserver started at http://127.19.68.62:39489/ using document root <none> and password file <none>
I20260810 19:29:50.446655 15577 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:50.446784 15577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:50.453298 15577 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 19:29:50.457367 15602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:50.459268 15577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:29:50.459563 15577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2886c323f59649058c4b1c3ae9c13ed4"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "10f93c2ce6a200064b02482a76fe75dc"
server_key_iv: "1300cf7d87c419fcb8b367e850627968"
server_key_version: "encryptionkey@0"
I20260810 19:29:50.461330 15577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:50.474807 15577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:50.479096 15577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:50.479372 15577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:50.498275 15577 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:36725
I20260810 19:29:50.498349 15693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:36725 every 8 connection(s)
I20260810 19:29:50.499845 15577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:29:50.502667 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 15577
I20260810 19:29:50.503573 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:33981
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=40517
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:29:50.509657 15695 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:29:50.514530 15695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap starting.
I20260810 19:29:50.537830 15695 log.cc:824] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:50.550590 15695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:29:50.551134 15695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap complete.
I20260810 19:29:50.560497 15695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:50.561393 15695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Initialized, Role: FOLLOWER
I20260810 19:29:50.562052 15695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:50.562287 15695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:29:50.562464 15695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:29:50.562628 15695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:29:50.566831 15695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:50.567379 15695 leader_election.cc:302] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886c323f59649058c4b1c3ae9c13ed4; no voters:
I20260810 19:29:50.567975 15695 leader_election.cc:288] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:29:50.568204 15701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:29:50.575126 15701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 LEADER]: Becoming Leader. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Running, Role: LEADER
I20260810 19:29:50.575773 15701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:29:50.576508 15695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:29:50.583091 15702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:29:50.583468 15702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:50.583858 15704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886c323f59649058c4b1c3ae9c13ed4. Latest consensus state: current_term: 2 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:29:50.584214 15704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:29:50.585649 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:29:50.594362 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:29:50.598495 15715 catalog_manager.cc:1259] Loaded cluster ID: 74e218db8f4349589f6d788e01aeab77
I20260810 19:29:50.598645 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:29:50.617062 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:29:50.619539 15715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Loaded TSK: 0
I20260810 19:29:50.662397 15715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Generated new TSK 1
I20260810 19:29:50.676973 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:29:50.929555 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:50.929824 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:50.929955 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:50.937230 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:50.937345 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:50.937492 15697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:29:50.947805 15697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:33981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=40517
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:50.949079 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:50.950446 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:50.965063 15742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:50.965163 15748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:50.965776 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:50.966760 15697 server_base.cc:1034] running on GCE node
I20260810 19:29:50.967443 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:50.968451 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:50.969687 15697 hybrid_clock.cc:648] HybridClock initialized: now 1786390190969614 us; error 61 us; skew 500 ppm
I20260810 19:29:50.970017 15697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:50 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390190, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:50.973316 15697 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:29:50.974778 15697 webserver.cc:460] Webserver started at http://127.19.68.1:40517/ using document root <none> and password file <none>
I20260810 19:29:50.975433 15697 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:50.975570 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:50.980942 15697 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:50.983913 15762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:50.985303 15697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:29:50.985448 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "4e9299fca4b8491f9be4a1a7252a772c"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "2c1eb5354697282a6235603f4546b02a"
server_key_iv: "8e3ce12ec75af5644d3af12f7d8ddb98"
server_key_version: "encryptionkey@0"
I20260810 19:29:50.986423 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:50.998324 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:51.002345 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:51.002578 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:51.003552 15697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:51.004945 15697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:51.005051 15697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.005178 15697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:51.005267 15697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.039963 15697 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:33981
I20260810 19:29:51.040035 15946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:33981 every 8 connection(s)
I20260810 19:29:51.041435 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:29:51.046177 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 15697
I20260810 19:29:51.046861 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:41029
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=39203
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:29:51 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390190, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:51.066355 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.044582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:58541)
0810 19:29:51.045530 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:51.045593 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:51.047523 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:51.049064 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:51.049072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:51.049648 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:51.050210 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.050235 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.052383 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.052387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.053774 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.053789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.055051 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.055057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.055254 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.055924 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:51.055950 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:51.058396 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:51.061821 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.061830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.061845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.062399 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.062815 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.062819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.062822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.063083 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.063808 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:51.063816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:51.064208 (+ 392us) client_negotiation.cc:769] Sending connection context
0810 19:29:51.064552 (+ 344us) client_negotiation.cc:240] Negotiation successful
0810 19:29:51.065147 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":410,"threads_started":1}
I20260810 19:29:51.066403 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.044324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58541 (local address 127.19.68.62:36725)
0810 19:29:51.044871 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:51.044883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:51.045731 (+ 848us) server_negotiation.cc:407] Connection header received
0810 19:29:51.048082 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:51.048099 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:51.048542 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:51.048913 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:51.050481 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.052170 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.053925 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.054813 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.055003 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.058694 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:51.058704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:51.058723 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:51.058772 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:51.061669 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.062538 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.062545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.062550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.062706 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.063202 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.063206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.063208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.063659 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:51.063813 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:51.064786 (+ 973us) server_negotiation.cc:299] Negotiation successful
0810 19:29:51.065209 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260810 19:29:51.069242 15947 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:51.069545 15947 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:51.070226 15947 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:51.072327 15634 ts_manager.cc:194] Registered new tserver with Master: 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1:33981)
I20260810 19:29:51.073853 15634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:58541
W20260810 19:29:51.396306 15953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:51.396579 15953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:51.396714 15953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:51.403998 15953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:51.404125 15953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:51.404284 15953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:29:51.414214 15953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:41029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=39203
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:51.415493 15953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:51.416873 15953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:51.429601 15963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:51.429631 15960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:51.429893 15959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:51.430529 15953 server_base.cc:1034] running on GCE node
I20260810 19:29:51.431125 15953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:51.432044 15953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:51.433203 15953 hybrid_clock.cc:648] HybridClock initialized: now 1786390191433137 us; error 51 us; skew 500 ppm
I20260810 19:29:51.433519 15953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:51 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390191, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:51.436652 15953 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:29:51.438129 15953 webserver.cc:460] Webserver started at http://127.19.68.2:39203/ using document root <none> and password file <none>
I20260810 19:29:51.438789 15953 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:51.438926 15953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:51.444406 15953 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 19:29:51.447340 15983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.448680 15953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:29:51.448863 15953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "3e423b694aec401583ab098207086c8d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "d2ac411e2f507b62814f7690951ccf51"
server_key_iv: "be817cdc86b867f365e6984548f5568b"
server_key_version: "encryptionkey@0"
I20260810 19:29:51.449919 15953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:51.470476 15953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:51.474534 15953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:51.474793 15953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:51.475754 15953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:51.477195 15953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:51.477300 15953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.477412 15953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:51.477480 15953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.509763 15953 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:41029
I20260810 19:29:51.509797 16165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:41029 every 8 connection(s)
I20260810 19:29:51.511293 15953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:29:51.521241 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 15953
I20260810 19:29:51.522102 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:38007
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=44297
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:29:51 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390191, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:51.531526 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.514005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:58937 (local address 127.19.68.62:36725)
0810 19:29:51.514214 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:51.514221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:51.515445 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 19:29:51.517358 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:51.517363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:51.517444 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:51.517530 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:51.518991 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.519777 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.521352 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.522031 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.522191 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.525819 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:51.525824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:51.525828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:51.525860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:51.528382 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.529299 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.529305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.529308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.529439 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.529918 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.529921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.529924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.530202 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:51.530329 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:51.531120 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 19:29:51.531296 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:29:51.532896 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.514488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:58937)
0810 19:29:51.515251 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:51.515319 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:51.517055 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:51.517713 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:51.517720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:51.518212 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:51.518802 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.518818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.519940 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.519944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.521202 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.521214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.522329 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.522339 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.522558 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.523185 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:51.523211 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:51.525565 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:51.528532 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.528541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.528555 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.529159 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.529545 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.529550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.529553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.529811 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.530328 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:51.530336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:51.530746 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 19:29:51.531048 (+ 302us) client_negotiation.cc:240] Negotiation successful
0810 19:29:51.531545 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":293,"threads_started":1}
I20260810 19:29:51.534396 16167 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:51.534679 16167 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:51.535486 16167 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:51.537118 15634 ts_manager.cc:194] Registered new tserver with Master: 3e423b694aec401583ab098207086c8d (127.19.68.2:41029)
I20260810 19:29:51.538152 15634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:58937
W20260810 19:29:51.857527 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:29:51.857811 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:29:51.857918 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:29:51.865401 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:29:51.865540 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:29:51.865672 16170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:29:51.875612 16170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:38007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=44297
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:29:51.876943 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:29:51.878422 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:29:51.891601 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:51.891634 16178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:29:51.891564 16179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:29:51.892099 16170 server_base.cc:1034] running on GCE node
I20260810 19:29:51.892758 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:29:51.893677 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:29:51.894876 16170 hybrid_clock.cc:648] HybridClock initialized: now 1786390191894808 us; error 45 us; skew 500 ppm
I20260810 19:29:51.895160 16170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:29:51 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390191, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:29:51.898208 16170 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:29:51.899573 16170 webserver.cc:460] Webserver started at http://127.19.68.3:44297/ using document root <none> and password file <none>
I20260810 19:29:51.900220 16170 fs_manager.cc:359] Metadata directory not provided
I20260810 19:29:51.900363 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:29:51.905541 16170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:29:51.908135 16192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.909428 16170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:29:51.909567 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "c57bcbf6a3a2415750f0e6bcb68b90ff"
server_key_iv: "857015dcf636a49969b768d389a2b732"
server_key_version: "encryptionkey@0"
I20260810 19:29:51.910579 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:29:51.921393 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:29:51.925344 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:29:51.925587 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:29:51.926542 16170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:29:51.927798 16170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:29:51.927906 16170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.928011 16170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:29:51.928081 16170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:29:51.958914 16170 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:38007
I20260810 19:29:51.958981 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:38007 every 8 connection(s)
I20260810 19:29:51.960338 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:29:51.964846 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 16170
Aug 10 19:29:51 dist-test-slave-2j7r krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390191, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:29:51.981328 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.19.68.62:36725)
0810 19:29:51.966179 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:51.966183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:51.967038 (+ 855us) server_negotiation.cc:407] Connection header received
0810 19:29:51.967167 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:51.967171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:51.967240 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:51.967345 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:51.968872 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.969838 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.971647 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.972313 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.972422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.975657 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:51.975662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:51.975665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:51.975697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:51.978653 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.979323 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.979327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.979329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.979443 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.979844 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.979848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.979850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.980119 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:51.980313 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:51.980699 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 19:29:51.980920 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
I20260810 19:29:51.982354 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.962912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:40035 (local address 127.19.68.62:36725)
0810 19:29:51.963136 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:51.963141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:51.964202 (+ 1061us) server_negotiation.cc:407] Connection header received
0810 19:29:51.966500 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:51.966504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:51.966569 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:51.966666 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:51.968396 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.969364 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.971470 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.972171 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.972354 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.976260 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:51.976266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:51.976270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:51.976308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:51.979098 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.979906 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.979912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.979916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.980038 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:51.980521 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:51.980525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:51.980527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:51.980794 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:51.980925 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:51.981840 (+ 915us) server_negotiation.cc:299] Negotiation successful
0810 19:29:51.982033 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":88}
I20260810 19:29:51.983578 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.963331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:40035)
0810 19:29:51.964047 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:51.964086 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:51.966127 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:51.966933 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:51.966941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:51.967650 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:29:51.968200 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.968218 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.969523 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.969527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.971296 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:51.971313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.972450 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:51.972459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:51.972745 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:51.973458 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:29:51.973488 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:29:51.976080 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:29:51.979232 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.979240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.979256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.979804 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.980142 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:29:51.980147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:29:51.980151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:29:51.980387 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:29:51.980993 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:29:51.981001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:29:51.981452 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 19:29:51.981764 (+ 312us) client_negotiation.cc:240] Negotiation successful
0810 19:29:51.982305 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
I20260810 19:29:51.984975 16386 heartbeater.cc:347] Connected to a master server at 127.19.68.62:36725
I20260810 19:29:51.985311 16386 heartbeater.cc:464] Registering TS with master...
I20260810 19:29:51.986037 16386 heartbeater.cc:511] Master 127.19.68.62:36725 requested a full tablet report, sending...
I20260810 19:29:51.987401 15634 ts_manager.cc:194] Registered new tserver with Master: 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007)
I20260810 19:29:51.988217 15634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:40035
I20260810 19:29:51.996306 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:29:52.002539 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:51.997186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.19.68.62:36725)
0810 19:29:51.997386 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:51.997393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:51.997772 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:29:51.997943 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:51.997947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:51.998072 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:51.998187 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:51.998557 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:51.999351 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.000716 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.001183 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.001311 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.001802 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:52.002020 (+ 218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 19:29:52.002323 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60336: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 19:29:52.002683 15950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60336: token expired
W20260810 19:29:52.003213 15512 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.68.62:36725)
I20260810 19:29:52.016062 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.003734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.19.68.62:36725)
0810 19:29:52.003927 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.003933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.003949 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:29:52.003986 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.003989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.004040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.004116 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:52.005561 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.006252 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.007763 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.008176 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.008286 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.010888 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:52.010900 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:52.010904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:52.010958 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:52.013505 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.014174 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.014179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.014181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.014318 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.014680 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.014684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.014686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.014993 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:52.015229 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.015540 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.015774 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:29:52.019632 15634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:29:52.027154 15634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:29:52.064965 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.049681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.19.68.1:33981)
0810 19:29:52.053469 (+ 3788us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.053478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.057207 (+ 3729us) server_negotiation.cc:407] Connection header received
0810 19:29:52.057313 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.057320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.057537 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.057780 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:52.058408 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.059933 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.062354 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.063867 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.063978 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.064215 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.064486 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.064649 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3596,"thread_start_us":3429,"threads_started":1}
I20260810 19:29:52.065121 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.049571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41676)
0810 19:29:52.056598 (+ 7027us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.056617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:52.056728 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:52.057783 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:52.057788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:52.057815 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:52.058198 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.058209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.060095 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.060100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.062206 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.062222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.064069 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.064076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.064273 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.064322 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 19:29:52.064429 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 19:29:52.064634 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6882,"mutex_wait_us":224,"thread_start_us":7428,"threads_started":1}
I20260810 19:29:52.065985 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.038311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38007 (local address 127.0.0.1:45592)
0810 19:29:52.055837 (+ 17526us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.055885 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:52.056062 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:52.058168 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:52.058173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:52.058201 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:52.058526 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.058535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.060479 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.060488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.062791 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.062800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.064939 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.064945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.065190 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.065229 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:29:52.065331 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 19:29:52.065469 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17239,"mutex_wait_us":82,"thread_start_us":17088,"threads_started":1}
I20260810 19:29:52.066119 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.038435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:41029 (local address 127.0.0.1:57112)
0810 19:29:52.056518 (+ 18083us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.056539 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:52.056686 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:52.057567 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:52.057574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:52.057604 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:52.058050 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.058062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.060093 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.060098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.062760 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.062777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.064921 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.064928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.065271 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.065299 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:29:52.065365 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 19:29:52.065472 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17934,"mutex_wait_us":243,"thread_start_us":17731,"threads_started":1}
I20260810 19:29:52.066450 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.039328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.19.68.2:41029)
0810 19:29:52.047526 (+ 8198us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.047532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.056822 (+ 9290us) server_negotiation.cc:407] Connection header received
0810 19:29:52.056939 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.056948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.057260 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.057459 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:52.058302 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.059953 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.062951 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.064387 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.064510 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.064699 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.065632 (+ 933us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.065829 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8062,"thread_start_us":7941,"threads_started":1}
I20260810 19:29:52.066892 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.039954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.19.68.3:38007)
0810 19:29:52.057645 (+ 17691us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.057653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.057686 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:29:52.057779 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.057786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.057992 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.058151 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:52.058684 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.060341 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.062951 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.064393 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.064510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.064729 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.066343 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.066505 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17475,"thread_start_us":17306,"threads_started":1}
I20260810 19:29:52.069633 16050 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 (DEFAULT_TABLE table=test-table [id=1161b8d5791e43a7ba5f2e29084e3c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:52.069587 15838 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 (DEFAULT_TABLE table=test-table [id=1161b8d5791e43a7ba5f2e29084e3c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:52.070747 16050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2bf18f4dd54bd39d1d439f0a585376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:52.070302 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 (DEFAULT_TABLE table=test-table [id=1161b8d5791e43a7ba5f2e29084e3c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:29:52.071046 15838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2bf18f4dd54bd39d1d439f0a585376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:52.071559 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2bf18f4dd54bd39d1d439f0a585376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:29:52.076872 15947 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:52.083287 16404 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Bootstrap starting.
I20260810 19:29:52.086222 16405 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Bootstrap starting.
I20260810 19:29:52.087692 16406 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Bootstrap starting.
I20260810 19:29:52.088142 16404 tablet_bootstrap.cc:654] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:52.089452 16404 log.cc:824] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:52.091998 16405 tablet_bootstrap.cc:654] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:52.091995 16406 tablet_bootstrap.cc:654] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Neither blocks nor log segments found. Creating new log.
I20260810 19:29:52.092293 16404 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: No bootstrap required, opened a new log
I20260810 19:29:52.092553 16404 ts_tablet_manager.cc:1397] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 19:29:52.093184 16405 log.cc:824] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:52.093184 16406 log.cc:824] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:29:52.096555 16405 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: No bootstrap required, opened a new log
I20260810 19:29:52.096834 16405 ts_tablet_manager.cc:1397] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 19:29:52.097168 16406 tablet_bootstrap.cc:492] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: No bootstrap required, opened a new log
I20260810 19:29:52.097440 16406 ts_tablet_manager.cc:1397] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260810 19:29:52.100980 16404 raft_consensus.cc:348] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.101373 16404 raft_consensus.cc:374] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:52.101490 16404 raft_consensus.cc:729] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e423b694aec401583ab098207086c8d, State: Initialized, Role: FOLLOWER
I20260810 19:29:52.102049 16404 consensus_queue.cc:260] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.103164 16167 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:52.103904 16404 ts_tablet_manager.cc:1428] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260810 19:29:52.107385 16405 raft_consensus.cc:348] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.107538 16406 raft_consensus.cc:348] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.107832 16405 raft_consensus.cc:374] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:52.107900 16406 raft_consensus.cc:374] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:29:52.107939 16405 raft_consensus.cc:729] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8cce502d574cf6bcdb79254ebbab9d, State: Initialized, Role: FOLLOWER
I20260810 19:29:52.107990 16406 raft_consensus.cc:729] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9299fca4b8491f9be4a1a7252a772c, State: Initialized, Role: FOLLOWER
I20260810 19:29:52.108532 16405 consensus_queue.cc:260] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.108536 16406 consensus_queue.cc:260] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.110181 16386 heartbeater.cc:503] Master 127.19.68.62:36725 was elected leader, sending a full tablet report...
I20260810 19:29:52.110471 16406 ts_tablet_manager.cc:1428] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260810 19:29:52.110836 16405 ts_tablet_manager.cc:1428] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260810 19:29:52.119053 16411 raft_consensus.cc:482] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:29:52.119314 16411 raft_consensus.cc:504] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.120827 16411 leader_election.cc:288] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007), 3e423b694aec401583ab098207086c8d (127.19.68.2:41029)
I20260810 19:29:52.129580 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.121294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:37705 (local address 127.19.68.3:38007)
0810 19:29:52.121495 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.121501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.122374 (+ 873us) server_negotiation.cc:407] Connection header received
0810 19:29:52.122479 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.122483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.122545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.122630 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:52.123224 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.124354 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.127169 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.128403 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.128531 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.128634 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.129168 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.129319 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:29:52.129580 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.121304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40993 (local address 127.19.68.2:41029)
0810 19:29:52.121496 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.121501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.122324 (+ 823us) server_negotiation.cc:407] Connection header received
0810 19:29:52.122784 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.122788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.122842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.122907 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:29:52.123486 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.124504 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.127169 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.128388 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.128531 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.128627 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.129168 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.129319 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:29:52.129837 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.121332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38007 (local address 127.19.68.1:37705)
0810 19:29:52.122211 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.122225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:52.122374 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:52.122645 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:52.122648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:52.122668 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:52.123005 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.123017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.124509 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.124513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.126990 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.127007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.128682 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.128688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.128908 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.128987 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 19:29:52.129080 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:29:52.129303 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":202,"spinlock_wait_cycles":41344,"thread_start_us":440,"threads_started":1}
I20260810 19:29:52.130119 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.121323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:41029 (local address 127.19.68.1:40993)
0810 19:29:52.122193 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.122218 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:29:52.122374 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:29:52.122960 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:29:52.122964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:29:52.122991 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:29:52.123326 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.123336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.124632 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.124637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.126990 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:29:52.127007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.128655 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.128666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:29:52.128907 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.128983 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 19:29:52.129080 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:29:52.129303 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":91,"spinlock_wait_cycles":15872,"thread_start_us":486,"threads_started":1}
I20260810 19:29:52.130831 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2bf18f4dd54bd39d1d439f0a585376" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" is_pre_election: true
I20260810 19:29:52.131047 16093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2bf18f4dd54bd39d1d439f0a585376" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e423b694aec401583ab098207086c8d" is_pre_election: true
I20260810 19:29:52.131289 16314 raft_consensus.cc:2454] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 0.
I20260810 19:29:52.131511 16093 raft_consensus.cc:2454] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 0.
I20260810 19:29:52.131896 15763 leader_election.cc:302] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d8cce502d574cf6bcdb79254ebbab9d, 4e9299fca4b8491f9be4a1a7252a772c; no voters:
I20260810 19:29:52.132246 16411 raft_consensus.cc:2781] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:29:52.132405 16411 raft_consensus.cc:482] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:29:52.132508 16411 raft_consensus.cc:3037] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:52.134687 16411 raft_consensus.cc:504] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.135284 16411 leader_election.cc:288] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 election: Requested vote from peers 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007), 3e423b694aec401583ab098207086c8d (127.19.68.2:41029)
I20260810 19:29:52.135658 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2bf18f4dd54bd39d1d439f0a585376" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
I20260810 19:29:52.135740 16093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2bf18f4dd54bd39d1d439f0a585376" candidate_uuid: "4e9299fca4b8491f9be4a1a7252a772c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e423b694aec401583ab098207086c8d"
I20260810 19:29:52.135864 16314 raft_consensus.cc:3037] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:52.135931 16093 raft_consensus.cc:3037] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:29:52.139042 16314 raft_consensus.cc:2454] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 1.
I20260810 19:29:52.139043 16093 raft_consensus.cc:2454] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9299fca4b8491f9be4a1a7252a772c in term 1.
I20260810 19:29:52.139446 15763 leader_election.cc:302] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d8cce502d574cf6bcdb79254ebbab9d, 4e9299fca4b8491f9be4a1a7252a772c; no voters:
I20260810 19:29:52.139735 16411 raft_consensus.cc:2781] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:29:52.140269 16411 raft_consensus.cc:686] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 LEADER]: Becoming Leader. State: Replica: 4e9299fca4b8491f9be4a1a7252a772c, State: Running, Role: LEADER
I20260810 19:29:52.140769 16411 consensus_queue.cc:237] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } }
I20260810 19:29:52.145876 15631 catalog_manager.cc:5495] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1). New cstate: current_term: 1 leader_uuid: "4e9299fca4b8491f9be4a1a7252a772c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9299fca4b8491f9be4a1a7252a772c" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33981 } health_report { overall_health: HEALTHY } } }
I20260810 19:29:52.166839 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.161235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.19.68.1:33981)
0810 19:29:52.161415 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.161419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.161436 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:29:52.161576 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.161580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.161660 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.161775 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:29:52.162184 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.162941 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.164554 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.165041 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.165213 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.165697 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:29:52.166237 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:29:52.166427 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.166521 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.166592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:29:52.174808 16093 raft_consensus.cc:1261] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Refusing update from remote peer 4e9299fca4b8491f9be4a1a7252a772c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:52.174788 16314 raft_consensus.cc:1261] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Refusing update from remote peer 4e9299fca4b8491f9be4a1a7252a772c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:29:52.175853 16423 consensus_queue.cc:1035] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e423b694aec401583ab098207086c8d" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:52.176198 16411 consensus_queue.cc:1035] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8cce502d574cf6bcdb79254ebbab9d" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:29:52.188201 16430 mvcc.cc:205] Tried to move back new op lower bound from 7317054227137769472 to 7317054227012927488. Current Snapshot: MvccSnapshot[applied={T|T < 7317054227137769472}]
I20260810 19:29:52.199635 15631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:52.199895 15631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:29:52.203270 15631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Sending DeleteTablet for 3 replicas of tablet 0e2bf18f4dd54bd39d1d439f0a585376
I20260810 19:29:52.204445 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:52 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:45592
I20260810 19:29:52.204458 16050 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:52 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260810 19:29:52.205268 15838 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2bf18f4dd54bd39d1d439f0a585376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:29:52 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:41676
I20260810 19:29:52.205689 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 15697
I20260810 19:29:52.206022 16443 tablet_replica.cc:330] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: stopping tablet replica
I20260810 19:29:52.205404 16445 tablet_replica.cc:330] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: stopping tablet replica
I20260810 19:29:52.207872 16445 raft_consensus.cc:2229] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:52.206470 16443 raft_consensus.cc:2229] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:29:52.208534 16443 raft_consensus.cc:2258] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:52.210363 16445 raft_consensus.cc:2258] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:52.212188 16452 tablet_replica.cc:330] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: stopping tablet replica
I20260810 19:29:52.212601 16452 raft_consensus.cc:2229] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:29:52.213299 16452 raft_consensus.cc:2258] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:29:52.213897 16443 ts_tablet_manager.cc:1905] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:52.215039 16452 ts_tablet_manager.cc:1905] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:52.220041 16445 ts_tablet_manager.cc:1905] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:29:52.221045 16443 ts_tablet_manager.cc:1918] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:52.221251 16443 log.cc:1196] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/0e2bf18f4dd54bd39d1d439f0a585376
I20260810 19:29:52.221840 16443 ts_tablet_manager.cc:1939] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4d8cce502d574cf6bcdb79254ebbab9d: Deleting consensus metadata
I20260810 19:29:52.223855 15604 catalog_manager.cc:4841] TS 4d8cce502d574cf6bcdb79254ebbab9d (127.19.68.3:38007): tablet 0e2bf18f4dd54bd39d1d439f0a585376 (table test-table [id=1161b8d5791e43a7ba5f2e29084e3c26]) successfully deleted
I20260810 19:29:52.225728 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.206567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.19.68.1:33981)
0810 19:29:52.207054 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.207058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.207072 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:29:52.207113 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.207116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.207171 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.207265 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:52.209010 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.210074 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.211625 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.212273 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.212390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.218464 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:52.218477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:52.218490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:52.218532 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:52.221694 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.222670 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.222679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.222683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.222857 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.223302 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.223307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.223310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.223737 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:52.224052 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.224829 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.225482 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260810 19:29:52.226526 15938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:52.231638 16445 ts_tablet_manager.cc:1918] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:52.231805 16445 log.cc:1196] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/0e2bf18f4dd54bd39d1d439f0a585376
I20260810 19:29:52.232214 16445 ts_tablet_manager.cc:1939] T 0e2bf18f4dd54bd39d1d439f0a585376 P 3e423b694aec401583ab098207086c8d: Deleting consensus metadata
I20260810 19:29:52.233757 15608 catalog_manager.cc:4841] TS 3e423b694aec401583ab098207086c8d (127.19.68.2:41029): tablet 0e2bf18f4dd54bd39d1d439f0a585376 (table test-table [id=1161b8d5791e43a7ba5f2e29084e3c26]) successfully deleted
I20260810 19:29:52.321647 16452 ts_tablet_manager.cc:1918] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:29:52.321979 16452 log.cc:1196] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/0e2bf18f4dd54bd39d1d439f0a585376
I20260810 19:29:52.322584 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 15697
W20260810 19:29:52.352800 15609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:33981 (error 108)
I20260810 19:29:52.352946 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 15953
I20260810 19:29:52.354743 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.354360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41700)
0810 19:29:52.354521 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.354608 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:29:52.354983 15609 catalog_manager.cc:4584] TS 4e9299fca4b8491f9be4a1a7252a772c (127.19.68.1:33981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e2bf18f4dd54bd39d1d439f0a585376: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
I20260810 19:29:52.370687 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.353870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.19.68.2:41029)
0810 19:29:52.354064 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.354071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.354112 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:29:52.354329 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.354334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.354401 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.354492 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:52.356124 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.356856 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.358222 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.358632 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.358742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.364857 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:52.364871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:52.364885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:52.364924 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:52.367893 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.368567 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.368574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.368576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.368706 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.369111 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.369117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.369120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.369524 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:52.369807 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.370161 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.370385 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:29:52.371382 16157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:52.408743 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.408264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41708)
0810 19:29:52.408473 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.408578 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 19:29:52.409869 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.409490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41718)
0810 19:29:52.409649 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.409746 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:29:52.460707 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 15953
I20260810 19:29:52.470558 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.470047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41720)
0810 19:29:52.470279 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.470379 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:29:52.471632 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.471307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41730)
0810 19:29:52.471447 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.471521 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:29:52.486325 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 16170
I20260810 19:29:52.499840 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.487118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.19.68.3:38007)
0810 19:29:52.487320 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:29:52.487326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:29:52.487347 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:29:52.487505 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:29:52.487512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:29:52.487584 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:29:52.487665 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:29:52.489016 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.489825 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.491110 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:29:52.491533 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:29:52.491645 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:29:52.494153 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:29:52.494168 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:29:52.494184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:29:52.494217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:29:52.497107 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.497755 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.497765 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.497769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.497957 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:29:52.498306 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:29:52.498311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:29:52.498315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:29:52.498673 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:29:52.498968 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 19:29:52.499293 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 19:29:52.499514 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:29:52.500520 16380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:52.552179 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.551683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41740)
0810 19:29:52.551924 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.552020 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 19:29:52.553464 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.552991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41754)
0810 19:29:52.553246 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.553352 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260810 19:29:52.587718 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 16170
I20260810 19:29:52.613070 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 15577
I20260810 19:29:52.614228 15686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:29:52.727160 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:29:52.726386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33981 (local address 127.0.0.1:41764)
0810 19:29:52.726782 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 19:29:52.726900 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260810 19:29:52.727203 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 15577
I20260810 19:30:12.756098 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--webserver_interface=127.19.68.62
--webserver_port=39489
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:13.099059 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:13.099462 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:13.099627 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:13.106913 16463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:30:13.107028 16463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:30:13.107084 16463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:30:13.107156 16463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:13.107230 16463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:13.107322 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:13.107385 16463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:13.107460 16463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:13.117352 16463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:13.118628 16463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:13.119982 16463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:13.131971 16469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:13.131891 16472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:13.131932 16477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:13.132768 16463 server_base.cc:1034] running on GCE node
I20260810 19:30:13.133597 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:13.135023 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:13.136231 16463 hybrid_clock.cc:648] HybridClock initialized: now 1786390213136184 us; error 30 us; skew 500 ppm
I20260810 19:30:13.136540 16463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:13 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:13.144731 16463 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:30:13.146378 16463 webserver.cc:460] Webserver started at http://127.19.68.62:39489/ using document root <none> and password file <none>
I20260810 19:30:13.147089 16463 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:13.147197 16463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:13.152678 16463 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:30:13.155546 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:13.156817 16463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:30:13.156972 16463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2886c323f59649058c4b1c3ae9c13ed4"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "10f93c2ce6a200064b02482a76fe75dc"
server_key_iv: "1300cf7d87c419fcb8b367e850627968"
server_key_version: "encryptionkey@0"
I20260810 19:30:13.157969 16463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:13.170025 16463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:13.173888 16463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:13.174139 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:13.192112 16463 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:36725
I20260810 19:30:13.192168 16568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:36725 every 8 connection(s)
I20260810 19:30:13.193591 16463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:13.198590 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 16463
I20260810 19:30:13.199644 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:33981
--local_ip_for_outbound_sockets=127.19.68.1
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=40517
--webserver_interface=127.19.68.1
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:30:13.201330 16569 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:30:13.205628 16569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap starting.
I20260810 19:30:13.227962 16569 log.cc:824] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:13.244514 16569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:30:13.245074 16569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Bootstrap complete.
I20260810 19:30:13.254642 16569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:30:13.255541 16569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Initialized, Role: FOLLOWER
I20260810 19:30:13.256208 16569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:30:13.256439 16569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:13.256559 16569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:13.256719 16569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:30:13.260557 16569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:30:13.261039 16569 leader_election.cc:302] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886c323f59649058c4b1c3ae9c13ed4; no voters:
I20260810 19:30:13.261739 16569 leader_election.cc:288] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:30:13.261879 16574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:30:13.263005 16574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [term 3 LEADER]: Becoming Leader. State: Replica: 2886c323f59649058c4b1c3ae9c13ed4, State: Running, Role: LEADER
I20260810 19:30:13.263542 16574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } }
I20260810 19:30:13.263942 16569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:13.266317 16575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:30:13.266535 16576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886c323f59649058c4b1c3ae9c13ed4. Latest consensus state: current_term: 3 leader_uuid: "2886c323f59649058c4b1c3ae9c13ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886c323f59649058c4b1c3ae9c13ed4" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36725 } } }
I20260810 19:30:13.267040 16575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:13.267055 16576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:13.267889 16585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:13.291599 16585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:13.293062 16585 catalog_manager.cc:1259] Loaded cluster ID: 74e218db8f4349589f6d788e01aeab77
I20260810 19:30:13.293238 16585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:13.297268 16585 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:13.298816 16585 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Loaded TSK: 0
I20260810 19:30:13.298964 16585 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2886c323f59649058c4b1c3ae9c13ed4: Loaded TSK: 1
F20260810 19:30:13.300035 16585 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f66d4ab39ad google::LogMessage::Fail() at ??:0
@ 0x7f66d4ab688a google::LogMessage::SendToLog() at ??:0
@ 0x7f66d4ab34de google::LogMessage::Flush() at ??:0
@ 0x7f66d4ab4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f66d88177d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f66d880673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f66ea4e21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f66ea4e1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f66ea56d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f66ea56d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f66ea5b75e9 std::function<>::operator()() at ??:0
@ 0x7f66ea4e5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f66ea4e3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66ea56b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f66ea56b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66d76ff345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f66d770291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f66d7702799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66d76ddeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f66da9b76db start_thread at ??:0
@ 0x7f66d394d61f clone at ??:0
W20260810 19:30:13.582916 16571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:13.583223 16571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:13.583385 16571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:13.591040 16571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:13.591168 16571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:13.591334 16571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:13.601310 16571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:33981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=40517
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:13.602581 16571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:13.603981 16571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:13.616858 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:13.616912 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:13.616951 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:13.617697 16571 server_base.cc:1034] running on GCE node
I20260810 19:30:13.618278 16571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:13.619199 16571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:13.620401 16571 hybrid_clock.cc:648] HybridClock initialized: now 1786390213620352 us; error 46 us; skew 500 ppm
I20260810 19:30:13.620707 16571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:13 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:13.623711 16571 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:30:13.625144 16571 webserver.cc:460] Webserver started at http://127.19.68.1:40517/ using document root <none> and password file <none>
I20260810 19:30:13.625780 16571 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:13.625900 16571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:13.631290 16571 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:30:13.634092 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:13.635354 16571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:30:13.635511 16571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "4e9299fca4b8491f9be4a1a7252a772c"
format_stamp: "Formatted at 2026-08-10 19:29:27 on dist-test-slave-2j7r"
server_key: "2c1eb5354697282a6235603f4546b02a"
server_key_iv: "8e3ce12ec75af5644d3af12f7d8ddb98"
server_key_version: "encryptionkey@0"
I20260810 19:30:13.636451 16571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:13.650360 16571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:13.654150 16571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:13.654381 16571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:13.655301 16571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:13.656991 16672 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 19:30:13.662122 16672 ts_tablet_manager.cc:1852] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 19:30:13.663056 16672 ts_tablet_manager.cc:1905] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:13.670006 16672 ts_tablet_manager.cc:1918] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:13.670234 16672 ts_tablet_manager.cc:1939] T 0e2bf18f4dd54bd39d1d439f0a585376 P 4e9299fca4b8491f9be4a1a7252a772c: Deleting consensus metadata
I20260810 19:30:13.670610 16571 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 19:30:13.670717 16571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260810 19:30:13.670832 16571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:13.670902 16571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:13.702915 16571 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:33981
I20260810 19:30:13.702991 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:33981 every 8 connection(s)
I20260810 19:30:13.704198 16571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:13.708967 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:13.707118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:49029)
0810 19:30:13.707859 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:13.708085 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":305,"threads_started":1}
W20260810 19:30:13.709575 16841 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:36725 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:36725: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
I20260810 19:30:13.710341 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:13.710026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:44363)
0810 19:30:13.710147 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:13.710212 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 19:30:13.711124 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:13.710815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:34789)
0810 19:30:13.710944 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:13.711013 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:30:13.711333 16841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:30:13.712929 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 16571
I20260810 19:30:13.713590 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:41029
--local_ip_for_outbound_sockets=127.19.68.2
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=39203
--webserver_interface=127.19.68.2
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:14.044433 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:14.044766 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:14.044929 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:14.052500 16845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:14.052632 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:14.052804 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:14.063122 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:41029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=39203
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:14.064417 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:14.065784 16845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:14.078785 16852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:14.078817 16854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:14.078846 16851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:14.079610 16845 server_base.cc:1034] running on GCE node
I20260810 19:30:14.080173 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:14.081075 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:14.082309 16845 hybrid_clock.cc:648] HybridClock initialized: now 1786390214082257 us; error 44 us; skew 500 ppm
I20260810 19:30:14.082623 16845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:14 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:14.085649 16845 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:30:14.087095 16845 webserver.cc:460] Webserver started at http://127.19.68.2:39203/ using document root <none> and password file <none>
I20260810 19:30:14.087741 16845 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:14.087858 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:14.093617 16845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:30:14.096282 16862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.097611 16845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:14.097769 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "3e423b694aec401583ab098207086c8d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "d2ac411e2f507b62814f7690951ccf51"
server_key_iv: "be817cdc86b867f365e6984548f5568b"
server_key_version: "encryptionkey@0"
I20260810 19:30:14.098697 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:14.110316 16845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:14.114445 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:14.114717 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:14.115697 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:14.117022 16845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:14.117147 16845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.117262 16845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:14.117336 16845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.149402 16845 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:41029
I20260810 19:30:14.149470 17044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:41029 every 8 connection(s)
I20260810 19:30:14.150840 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:14.155956 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.153721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:39051)
0810 19:30:14.154576 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.154877 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":320,"threads_started":1}
I20260810 19:30:14.156134 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 16845
W20260810 19:30:14.156692 17046 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:36725 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:36725: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
I20260810 19:30:14.156910 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:38007
--local_ip_for_outbound_sockets=127.19.68.3
--tserver_master_addrs=127.19.68.62:36725
--webserver_port=44297
--webserver_interface=127.19.68.3
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:30:14.157610 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.157252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:48795)
0810 19:30:14.157407 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.157488 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:30:14.158468 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.158102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:52511)
0810 19:30:14.158241 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.158334 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:30:14.158703 17046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:30:14.494971 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:14.495231 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:14.495337 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:14.502481 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:14.502590 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:14.502740 17052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:14.512336 17052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:38007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=44297
--tserver_master_addrs=127.19.68.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:14.513578 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:14.514909 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:14.527364 17063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:14.527397 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:14.527460 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:14.528170 17052 server_base.cc:1034] running on GCE node
I20260810 19:30:14.528694 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:14.529592 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:14.530772 17052 hybrid_clock.cc:648] HybridClock initialized: now 1786390214530719 us; error 48 us; skew 500 ppm
I20260810 19:30:14.531050 17052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:14 dist-test-slave-2j7r krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:14.534024 17052 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:30:14.535349 17052 webserver.cc:460] Webserver started at http://127.19.68.3:44297/ using document root <none> and password file <none>
I20260810 19:30:14.535995 17052 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:14.536114 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:14.541349 17052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:14.543912 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.545272 17052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:30:14.545424 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "4d8cce502d574cf6bcdb79254ebbab9d"
format_stamp: "Formatted at 2026-08-10 19:29:28 on dist-test-slave-2j7r"
server_key: "c57bcbf6a3a2415750f0e6bcb68b90ff"
server_key_iv: "857015dcf636a49969b768d389a2b732"
server_key_version: "encryptionkey@0"
I20260810 19:30:14.546422 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:14.567359 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:14.571240 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:14.571492 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:14.572467 17052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:14.573849 17052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:14.573954 17052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.574069 17052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:14.574148 17052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:14.604321 17052 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:38007
I20260810 19:30:14.604398 17269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:38007 every 8 connection(s)
I20260810 19:30:14.605778 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:14.609519 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 17052
I20260810 19:30:14.610872 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.608712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:34363)
0810 19:30:14.609599 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.609891 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":334,"threads_started":1}
W20260810 19:30:14.611538 17270 heartbeater.cc:644] Failed to heartbeat to 127.19.68.62:36725 (0 consecutive failures): Network error: Failed to ping master at 127.19.68.62:36725: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
I20260810 19:30:14.612318 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.612002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:43515)
0810 19:30:14.612148 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.612212 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 19:30:14.613193 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.612809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:37237)
0810 19:30:14.612964 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.613034 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 19:30:14.613476 17270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:30:14.712587 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:14.711832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:37611)
0810 19:30:14.712279 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:14.712357 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":211,"threads_started":1}
I20260810 19:30:15.159893 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:15.159164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:41987)
0810 19:30:15.159596 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:15.159671 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":208,"threads_started":1}
I20260810 19:30:15.614704 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:15.613969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:57483)
0810 19:30:15.614398 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:15.614483 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260810 19:30:15.714339 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:15.713563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:54979)
0810 19:30:15.714041 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:15.714128 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
I20260810 19:30:16.161478 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:16.160690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:47701)
0810 19:30:16.161210 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:16.161293 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
I20260810 19:30:16.616231 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:16.615533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:57527)
0810 19:30:16.615988 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:16.616066 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
I20260810 19:30:16.715906 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:16.715160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:51281)
0810 19:30:16.715635 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:16.715726 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":219,"threads_started":1}
I20260810 19:30:17.163060 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:17.162335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:36267)
0810 19:30:17.162777 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:17.162860 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260810 19:30:17.617825 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:17.617061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:44107)
0810 19:30:17.617552 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:17.617636 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260810 19:30:17.717679 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:17.716822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:36065)
0810 19:30:17.717350 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:17.717462 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":245,"threads_started":1}
I20260810 19:30:18.164824 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:18.163971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:52881)
0810 19:30:18.164490 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:18.164580 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":259,"threads_started":1}
I20260810 19:30:18.619472 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:18.618707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:44059)
0810 19:30:18.619186 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:18.619278 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260810 19:30:18.719252 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:18.718576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:59001)
0810 19:30:18.719016 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:18.719098 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260810 19:30:19.166666 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:19.165859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:52133)
0810 19:30:19.166411 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:19.166493 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":290,"threads_started":1}
I20260810 19:30:19.621011 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:19.620304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:32977)
0810 19:30:19.620763 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:19.620842 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
I20260810 19:30:19.720917 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:19.720147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:50157)
0810 19:30:19.720640 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:19.720723 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260810 19:30:20.168457 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:20.167494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:57849)
0810 19:30:20.168052 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:20.168158 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":292,"threads_started":1}
I20260810 19:30:20.622649 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:20.621873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:49951)
0810 19:30:20.622393 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:20.622486 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":252,"threads_started":1}
I20260810 19:30:20.722723 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:20.721868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:38699)
0810 19:30:20.722426 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:20.722513 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":247,"threads_started":1}
I20260810 19:30:21.170220 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:21.169410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:40225)
0810 19:30:21.169914 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:21.169999 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":206,"threads_started":1}
I20260810 19:30:21.624353 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:21.623545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:54219)
0810 19:30:21.624054 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:21.624135 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":203,"threads_started":1}
I20260810 19:30:21.724341 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:21.723635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:38397)
0810 19:30:21.724074 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:21.724158 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260810 19:30:22.171775 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:22.171045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:36225)
0810 19:30:22.171518 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:22.171600 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260810 19:30:22.625941 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:22.625246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:40799)
0810 19:30:22.625688 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:22.625777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260810 19:30:22.725940 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:22.725202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:60197)
0810 19:30:22.725677 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:22.725776 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":221,"threads_started":1}
I20260810 19:30:23.173473 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:23.172704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:34671)
0810 19:30:23.173209 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:23.173292 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":235,"threads_started":1}
I20260810 19:30:23.627488 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:23.626741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:52537)
0810 19:30:23.627218 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:23.627307 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
I20260810 19:30:23.727648 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:23.726857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:44815)
0810 19:30:23.727365 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:23.727456 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
I20260810 19:30:24.175019 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:24.174338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:46893)
0810 19:30:24.174785 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:24.174862 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260810 19:30:24.629202 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:24.628416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:50823)
0810 19:30:24.628874 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:24.628954 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260810 19:30:24.729301 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:24.728522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:48901)
0810 19:30:24.728980 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:24.729072 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260810 19:30:25.176654 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:25.175904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:53459)
0810 19:30:25.176388 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:25.176471 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
I20260810 19:30:25.630870 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:25.630105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:60263)
0810 19:30:25.630593 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:25.630681 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260810 19:30:25.730917 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:25.730196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:54885)
0810 19:30:25.730659 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:25.730741 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260810 19:30:26.178248 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:26.177538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:37753)
0810 19:30:26.178010 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:26.178094 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260810 19:30:26.632432 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:26.631679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:45423)
0810 19:30:26.632143 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:26.632226 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260810 19:30:26.732697 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:26.731888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:46957)
0810 19:30:26.732376 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:26.732509 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":247,"threads_started":1}
I20260810 19:30:27.179867 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:27.179099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:33577)
0810 19:30:27.179564 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:27.179684 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":214,"threads_started":1}
I20260810 19:30:27.634284 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:27.633412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:49275)
0810 19:30:27.633984 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:27.634102 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260810 19:30:27.734393 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:27.733617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:59717)
0810 19:30:27.734110 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:27.734197 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
I20260810 19:30:28.181677 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:28.180810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:58185)
0810 19:30:28.181373 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:28.181472 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":300,"threads_started":1}
I20260810 19:30:28.635848 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:28.635107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:48005)
0810 19:30:28.635560 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:28.635643 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260810 19:30:28.736286 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:28.735535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.1:42623)
0810 19:30:28.736022 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:28.736114 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260810 19:30:29.183499 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:29.182766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.2:43959)
0810 19:30:29.183195 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:29.183306 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":215,"threads_started":1}
I20260810 19:30:29.617720 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 16571
I20260810 19:30:29.636319 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:29.619019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.19.68.1:33981)
0810 19:30:29.619675 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:29.619690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:29.620454 (+ 764us) server_negotiation.cc:407] Connection header received
0810 19:30:29.622280 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:29.622292 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:29.622725 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:29.623034 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:29.624439 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.625805 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.626960 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.627592 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.627722 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:29.630366 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:29.630373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:29.630388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:29.630432 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:29.633316 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.633897 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.633904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.633910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.634077 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.634407 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.634412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.634414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.635014 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:29.635254 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:29.635565 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:30:29.635972 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":359,"threads_started":1}
I20260810 19:30:29.637789 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:29.636949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:36725 (local address 127.19.68.3:58011)
0810 19:30:29.637471 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:29.637617 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.62:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260810 19:30:29.637930 16831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:29.722973 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 16571
I20260810 19:30:29.754514 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 16845
I20260810 19:30:29.771742 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:29.755533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.19.68.2:41029)
0810 19:30:29.756114 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:29.756128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:29.756164 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:30:29.757787 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:29.757804 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:29.758234 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:29.758515 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:29.759856 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.761127 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.762347 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.763001 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.763137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:29.765743 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:29.765751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:29.765765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:29.765808 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:29.768711 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.769460 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.769466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.769473 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.769657 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.769975 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.769979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.769981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.770522 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:29.770772 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:29.771061 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 19:30:29.771450 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":287,"threads_started":1}
I20260810 19:30:29.773065 17034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:29.858355 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 16845
I20260810 19:30:29.881546 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 17052
I20260810 19:30:29.900599 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:29.882675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.19.68.3:38007)
0810 19:30:29.883394 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:29.883407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:29.883455 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:30:29.885281 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:29.885293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:29.885726 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:29.886091 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:29.887751 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.889047 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.890617 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:29.891311 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:29.891497 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:29.894362 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:29.894370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:29.894384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:29.894425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:29.897310 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.898045 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.898052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.898058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.898234 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:29.898636 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:29.898641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:29.898644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:29.899154 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:29.899466 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:29.899880 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 19:30:29.900291 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":326,"threads_started":1}
I20260810 19:30:29.902029 17262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:29.985723 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 17052
2026-08-10T19:30:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64631 ms)
[----------] 30 tests from SecurityITest (300238 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:30 dist-test-slave-2j7r krb5kdc[17374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:30 dist-test-slave-2j7r krb5kdc[17374](info): set up 2 sockets
Aug 10 19:30:30 dist-test-slave-2j7r krb5kdc[17374](info): commencing operation
krb5kdc: starting...
W20260810 19:30:31.266340 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:31 dist-test-slave-2j7r krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:30:31.578873 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34079
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:31.922544 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:31.922843 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:31.922992 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:31.930567 17394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:31.930699 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:31.930785 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:31.930871 17394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:31.930933 17394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:31.941660 17394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:31.943063 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:31.944487 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:31.957192 17404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:31.957242 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:31.957130 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:31.957952 17394 server_base.cc:1034] running on GCE node
I20260810 19:30:31.958710 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:31.960093 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:31.961294 17394 hybrid_clock.cc:648] HybridClock initialized: now 1786390231961252 us; error 38 us; skew 500 ppm
I20260810 19:30:31.961638 17394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:31 dist-test-slave-2j7r krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:31.964783 17394 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:30:31.966288 17394 webserver.cc:460] Webserver started at http://127.19.68.62:38141/ using document root <none> and password file <none>
I20260810 19:30:31.966945 17394 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:31.967048 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:31.967301 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:31.970331 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "161afe1821ad4bd3b79151885c4fc44a"
format_stamp: "Formatted at 2026-08-10 19:30:31 on dist-test-slave-2j7r"
server_key: "669ce4ff2448ac5b28ce24110812040e"
server_key_iv: "044d300ed3660bfcfa95d6546f449f4c"
server_key_version: "encryptionkey@0"
I20260810 19:30:31.971047 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "161afe1821ad4bd3b79151885c4fc44a"
format_stamp: "Formatted at 2026-08-10 19:30:31 on dist-test-slave-2j7r"
server_key: "669ce4ff2448ac5b28ce24110812040e"
server_key_iv: "044d300ed3660bfcfa95d6546f449f4c"
server_key_version: "encryptionkey@0"
I20260810 19:30:31.976258 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:30:31.979761 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:31.981051 17394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:30:31.981259 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "161afe1821ad4bd3b79151885c4fc44a"
format_stamp: "Formatted at 2026-08-10 19:30:31 on dist-test-slave-2j7r"
server_key: "669ce4ff2448ac5b28ce24110812040e"
server_key_iv: "044d300ed3660bfcfa95d6546f449f4c"
server_key_version: "encryptionkey@0"
I20260810 19:30:31.981441 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:32.013283 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:32.017799 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:32.018080 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:32.037514 17394 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:34079
I20260810 19:30:32.037563 17510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:34079 every 8 connection(s)
I20260810 19:30:32.038942 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:32.042650 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:32.046461 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 17394
I20260810 19:30:32.046680 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:32.047356 19728 external_mini_cluster.cc:1428] Setting key 4cb6ced50e62867102e40e3b22382e24
I20260810 19:30:32.054177 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: Bootstrap starting.
I20260810 19:30:32.057760 17514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:32.058908 17514 log.cc:824] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:32.061769 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: No bootstrap required, opened a new log
Aug 10 19:30:32 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:32.069646 17514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } }
I20260810 19:30:32.070093 17514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:32.070200 17514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161afe1821ad4bd3b79151885c4fc44a, State: Initialized, Role: FOLLOWER
I20260810 19:30:32.070753 17514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } }
I20260810 19:30:32.070976 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:32.049411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.19.68.62:34079)
0810 19:30:32.050171 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:32.050186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:32.050231 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:30:32.052500 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:32.052523 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:32.052899 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:32.053258 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:32.055113 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.056451 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.058285 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.058941 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.059101 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:32.063376 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:32.063384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:32.063398 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:32.063445 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:32.066653 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:32.067394 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:32.067401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:32.067408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:32.067591 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:32.068005 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:32.068008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:32.068010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:32.068576 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:32.068888 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:32.069369 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 19:30:32.069786 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":333,"threads_started":1}
I20260810 19:30:32.071002 17514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:32.071190 17514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:32.071388 17514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:32.074245 17514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } }
I20260810 19:30:32.074668 17514 leader_election.cc:302] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161afe1821ad4bd3b79151885c4fc44a; no voters:
I20260810 19:30:32.075166 17514 leader_election.cc:288] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:32.075309 17522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:32.076373 17522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [term 1 LEADER]: Becoming Leader. State: Replica: 161afe1821ad4bd3b79151885c4fc44a, State: Running, Role: LEADER
I20260810 19:30:32.076839 17522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } }
I20260810 19:30:32.077252 17514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:32.079663 17523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161afe1821ad4bd3b79151885c4fc44a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } } }
I20260810 19:30:32.079632 17524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161afe1821ad4bd3b79151885c4fc44a. Latest consensus state: current_term: 1 leader_uuid: "161afe1821ad4bd3b79151885c4fc44a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161afe1821ad4bd3b79151885c4fc44a" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34079 } } }
I20260810 19:30:32.080207 17523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:32.080214 17524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:32.081993 17531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:32.087397 17531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:32.097283 17531 catalog_manager.cc:1347] Generated new cluster ID: d60a7d70e19b4659aa3988ddd7431110
I20260810 19:30:32.097393 17531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:32.113209 17531 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:32.115074 17531 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:32.127300 17531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: Generated new TSK 0
I20260810 19:30:32.128088 17531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:30:32.208362 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:32.550081 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:32.550416 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:32.550560 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:32.557897 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:32.558014 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:32.558164 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:32.567884 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:32.569175 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:32.570508 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:32.583439 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:32.583479 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:32.584103 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:32.584252 17553 server_base.cc:1034] running on GCE node
I20260810 19:30:32.584892 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:32.585834 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:32.587042 17553 hybrid_clock.cc:648] HybridClock initialized: now 1786390232586996 us; error 39 us; skew 500 ppm
I20260810 19:30:32.587342 17553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:32 dist-test-slave-2j7r krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390232, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:32.593047 17553 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:30:32.594539 17553 webserver.cc:460] Webserver started at http://127.19.68.1:43257/ using document root <none> and password file <none>
I20260810 19:30:32.595187 17553 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:32.595316 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:32.595618 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:32.598577 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "5976931ac7a64a269557766adcd535aa"
format_stamp: "Formatted at 2026-08-10 19:30:32 on dist-test-slave-2j7r"
server_key: "6c948546b2aabc4cee080fd4dd46c2d0"
server_key_iv: "35e288e47a14bfe627214f3b73d1222d"
server_key_version: "encryptionkey@0"
I20260810 19:30:32.599316 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "5976931ac7a64a269557766adcd535aa"
format_stamp: "Formatted at 2026-08-10 19:30:32 on dist-test-slave-2j7r"
server_key: "6c948546b2aabc4cee080fd4dd46c2d0"
server_key_iv: "35e288e47a14bfe627214f3b73d1222d"
server_key_version: "encryptionkey@0"
I20260810 19:30:32.604239 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:30:32.607586 17592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:32.608834 17553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:30:32.608983 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "5976931ac7a64a269557766adcd535aa"
format_stamp: "Formatted at 2026-08-10 19:30:32 on dist-test-slave-2j7r"
server_key: "6c948546b2aabc4cee080fd4dd46c2d0"
server_key_iv: "35e288e47a14bfe627214f3b73d1222d"
server_key_version: "encryptionkey@0"
I20260810 19:30:32.609205 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:32.652894 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:32.657126 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:32.657450 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:32.658456 17553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:32.659878 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:32.659966 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:32.660102 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:32.660173 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:32.691761 17553 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:43493
I20260810 19:30:32.691808 17781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:43493 every 8 connection(s)
I20260810 19:30:32.693061 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:32.699645 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 17553
I20260810 19:30:32.699837 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:32.700487 19728 external_mini_cluster.cc:1428] Setting key 46beaf6c98809666c42225fef76ce8fa
Aug 10 19:30:32 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390232, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:32.712103 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:32.695560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:42301 (local address 127.19.68.62:34079)
0810 19:30:32.696020 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:32.696025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:32.696842 (+ 817us) server_negotiation.cc:407] Connection header received
0810 19:30:32.698728 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:32.698734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:32.698815 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:32.698912 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:32.700449 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.701320 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.702951 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.703351 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.703477 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:32.706797 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:32.706802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:32.706805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:32.706837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:32.709163 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:32.709849 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:32.709855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:32.709858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:32.709980 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:32.710392 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:32.710396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:32.710398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:32.710667 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:32.710799 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:32.711493 (+ 694us) server_negotiation.cc:299] Negotiation successful
0810 19:30:32.711726 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":236,"threads_started":1}
I20260810 19:30:32.713068 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:32.696023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34079 (local address 127.19.68.1:42301)
0810 19:30:32.696696 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:32.696734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:32.698463 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:32.699151 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:32.699159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:32.699654 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:32.700268 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:32.700286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.701468 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.701472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:32.702809 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:32.702822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:32.703576 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:32.703585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:32.703803 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:32.704459 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:32.704481 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:32.706594 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:32.709295 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:32.709304 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:32.709318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:32.709735 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:32.710088 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:32.710092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:32.710096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:32.710281 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:32.710792 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:32.710800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:32.711126 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 19:30:32.711357 (+ 231us) client_negotiation.cc:240] Negotiation successful
0810 19:30:32.711866 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260810 19:30:32.714592 17782 heartbeater.cc:347] Connected to a master server at 127.19.68.62:34079
I20260810 19:30:32.714865 17782 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:32.715550 17782 heartbeater.cc:511] Master 127.19.68.62:34079 requested a full tablet report, sending...
I20260810 19:30:32.717551 17450 ts_manager.cc:194] Registered new tserver with Master: 5976931ac7a64a269557766adcd535aa (127.19.68.1:43493)
I20260810 19:30:32.718961 17450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:42301
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:30:32.766491 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:33.105677 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:33.105999 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:33.106138 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:33.113652 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:33.113778 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:33.113940 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:33.123845 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:33.125205 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:33.126539 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:33.139570 17800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:33.139599 17804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:33.139729 17798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:33.140202 17790 server_base.cc:1034] running on GCE node
I20260810 19:30:33.140889 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:33.141892 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:33.143090 17790 hybrid_clock.cc:648] HybridClock initialized: now 1786390233143039 us; error 45 us; skew 500 ppm
I20260810 19:30:33.143396 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:33 dist-test-slave-2j7r krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390233, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:33.146607 17790 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:30:33.148118 17790 webserver.cc:460] Webserver started at http://127.19.68.2:39475/ using document root <none> and password file <none>
I20260810 19:30:33.148767 17790 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:33.148869 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:33.149187 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:33.152030 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "b75b0491ca604f70b3992abc53a72078"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "288fa664bd0e83e095894393a4845520"
server_key_iv: "144e334d81a2509e64cdaf79d7bf13b6"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.152730 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "b75b0491ca604f70b3992abc53a72078"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "288fa664bd0e83e095894393a4845520"
server_key_iv: "144e334d81a2509e64cdaf79d7bf13b6"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.157894 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:30:33.161280 17819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.162587 17790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:30:33.162727 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "b75b0491ca604f70b3992abc53a72078"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "288fa664bd0e83e095894393a4845520"
server_key_iv: "144e334d81a2509e64cdaf79d7bf13b6"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.162911 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:33.183367 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:33.187479 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:33.187740 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:33.188653 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:33.190019 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:33.190091 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.190212 17790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:33.190263 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.222383 17790 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:43199
I20260810 19:30:33.222442 18018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:43199 every 8 connection(s)
I20260810 19:30:33.223706 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:33.228242 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 17790
I20260810 19:30:33.228457 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:33.229009 19728 external_mini_cluster.cc:1428] Setting key 02a58c4e9724a9cabfa369b98eae7f0a
Aug 10 19:30:33 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390233, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:33.244727 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.226232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:60281 (local address 127.19.68.62:34079)
0810 19:30:33.226714 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.226719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.227571 (+ 852us) server_negotiation.cc:407] Connection header received
0810 19:30:33.230001 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.230005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.230079 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.230190 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:33.232004 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.232740 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.234826 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.235271 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.235362 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.239301 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:33.239306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:33.239309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:33.239342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:33.241872 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.242662 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.242665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.242668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.242793 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.243216 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.243219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.243221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.243459 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:33.243564 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.244292 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.244454 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":231,"threads_started":1}
I20260810 19:30:33.246002 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.226679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34079 (local address 127.19.68.2:60281)
0810 19:30:33.227414 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:33.227456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:33.229716 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:33.230399 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:33.230408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:33.231033 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:33.231821 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.231847 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.232893 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.232899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.234703 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.234719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.235495 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.235501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.235773 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.236465 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:33.236500 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:33.239074 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:33.242002 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:33.242011 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:33.242026 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:33.242566 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:33.242887 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:33.242892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:33.242895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:33.243126 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:33.243567 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:33.243574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:33.243963 (+ 389us) client_negotiation.cc:769] Sending connection context
0810 19:30:33.244240 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 19:30:33.244729 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":280,"threads_started":1}
I20260810 19:30:33.247484 18019 heartbeater.cc:347] Connected to a master server at 127.19.68.62:34079
I20260810 19:30:33.247768 18019 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:33.248484 18019 heartbeater.cc:511] Master 127.19.68.62:34079 requested a full tablet report, sending...
I20260810 19:30:33.249898 17450 ts_manager.cc:194] Registered new tserver with Master: b75b0491ca604f70b3992abc53a72078 (127.19.68.2:43199)
I20260810 19:30:33.250701 17450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:60281
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:30:33.295971 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:33.625329 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:33.625597 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:33.625736 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:33.632766 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:33.632884 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:33.633029 18032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:33.642468 18032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:33.643682 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:33.644975 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:33.657699 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:33.657732 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:33.657665 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:33.658488 18032 server_base.cc:1034] running on GCE node
I20260810 19:30:33.659039 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:33.659905 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:33.661123 18032 hybrid_clock.cc:648] HybridClock initialized: now 1786390233661060 us; error 29 us; skew 500 ppm
I20260810 19:30:33.661409 18032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:33 dist-test-slave-2j7r krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390233, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:33.664366 18032 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:30:33.665753 18032 webserver.cc:460] Webserver started at http://127.19.68.3:42125/ using document root <none> and password file <none>
I20260810 19:30:33.666438 18032 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:33.666540 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:33.666810 18032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:33.669693 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "28b5f678f28f4b27b6da6029dc13d9e0"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "4826c2ce84839a5675bc3d2b9766d9e1"
server_key_iv: "e21475d57de7a543e6357fb5b1ce8d8d"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.670413 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "28b5f678f28f4b27b6da6029dc13d9e0"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "4826c2ce84839a5675bc3d2b9766d9e1"
server_key_iv: "e21475d57de7a543e6357fb5b1ce8d8d"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.675415 18032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:30:33.678951 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.680303 18032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:30:33.680433 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "28b5f678f28f4b27b6da6029dc13d9e0"
format_stamp: "Formatted at 2026-08-10 19:30:33 on dist-test-slave-2j7r"
server_key: "4826c2ce84839a5675bc3d2b9766d9e1"
server_key_iv: "e21475d57de7a543e6357fb5b1ce8d8d"
server_key_version: "encryptionkey@0"
I20260810 19:30:33.680591 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:33.707777 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:33.711542 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:33.711789 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:33.712720 18032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:33.714157 18032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:33.714241 18032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.714341 18032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:33.714385 18032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:33.721869 17782 heartbeater.cc:503] Master 127.19.68.62:34079 was elected leader, sending a full tablet report...
I20260810 19:30:33.745206 18032 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:38581
I20260810 19:30:33.745289 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:38581 every 8 connection(s)
I20260810 19:30:33.746512 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:33.754315 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 18032
I20260810 19:30:33.754505 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:33.754958 19728 external_mini_cluster.cc:1428] Setting key 620ce8e4aea9b07c5f961701bd4cf3cb
Aug 10 19:30:33 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390233, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:33.765467 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.748990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:41661 (local address 127.19.68.62:34079)
0810 19:30:33.749440 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.749444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.750309 (+ 865us) server_negotiation.cc:407] Connection header received
0810 19:30:33.752169 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.752177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.752262 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.752359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:33.753867 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.754726 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.756538 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.756935 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.757044 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.760226 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:33.760231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:33.760235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:33.760273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:33.762588 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.763277 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.763280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.763283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.763404 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.763813 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.763816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.763818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.764070 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:33.764212 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.764845 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.765056 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":235,"threads_started":1}
I20260810 19:30:33.766316 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.749445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:34079 (local address 127.19.68.3:41661)
0810 19:30:33.750163 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:33.750200 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:33.751897 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:33.752597 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:33.752605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:33.753155 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:33.753701 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.753719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.754875 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.754879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.756401 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.756420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.757171 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.757179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.757402 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.758006 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:33.758027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:33.760039 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:33.762716 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:33.762724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:33.762738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:33.763154 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:33.763513 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:33.763518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:33.763522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:33.763698 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:33.764195 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:33.764202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:33.764495 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 19:30:33.764713 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:30:33.765235 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":289,"threads_started":1}
I20260810 19:30:33.767639 18253 heartbeater.cc:347] Connected to a master server at 127.19.68.62:34079
I20260810 19:30:33.767900 18253 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:33.768590 18253 heartbeater.cc:511] Master 127.19.68.62:34079 requested a full tablet report, sending...
I20260810 19:30:33.770219 17449 ts_manager.cc:194] Registered new tserver with Master: 28b5f678f28f4b27b6da6029dc13d9e0 (127.19.68.3:38581)
I20260810 19:30:33.771106 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:41661
I20260810 19:30:33.782516 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:33.799247 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.787060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.19.68.62:34079)
0810 19:30:33.787203 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.787207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.787661 (+ 454us) server_negotiation.cc:407] Connection header received
0810 19:30:33.787830 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.787834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.787896 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.787977 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:33.789493 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.790203 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.791555 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.791947 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.792051 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.794301 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:33.794305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:33.794308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:33.794338 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:33.796666 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.797345 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.797349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.797352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.797478 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:33.797846 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:33.797849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:33.797851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:33.798152 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:33.798380 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.798721 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.798937 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:30:33.803118 17449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:33.810061 17449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:33.848222 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.822335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.19.68.3:38581)
0810 19:30:33.835106 (+ 12771us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.835115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.839368 (+ 4253us) server_negotiation.cc:407] Connection header received
0810 19:30:33.839706 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.839714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.839992 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.840190 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:33.841281 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.842792 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.845555 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.847110 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.847228 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.847473 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.847734 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.847891 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12592,"thread_start_us":12404,"threads_started":1}
I20260810 19:30:33.848277 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.821998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38581 (local address 127.0.0.1:51454)
0810 19:30:33.839178 (+ 17180us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:33.839240 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:33.839478 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:33.840556 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:33.840564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:33.840611 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:33.841100 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.841118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.842978 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.842984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.845388 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.845404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.847305 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.847311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.847487 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.847536 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 19:30:33.847648 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 19:30:33.847840 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16900,"thread_start_us":18500,"threads_started":1}
I20260810 19:30:33.848515 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.822513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43493 (local address 127.0.0.1:52118)
0810 19:30:33.840055 (+ 17542us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:33.840072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:33.840178 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:33.841514 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:33.841517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:33.841536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:33.841987 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.842004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.843840 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.843846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.845531 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.845546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.847631 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.847637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.847820 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.847852 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:30:33.847928 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:30:33.848057 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17421,"mutex_wait_us":3,"spinlock_wait_cycles":1920,"thread_start_us":17258,"threads_started":1}
I20260810 19:30:33.849246 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.822769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.19.68.1:43493)
0810 19:30:33.840736 (+ 17967us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.840744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.840771 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:30:33.840879 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.840886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.841126 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.841308 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:33.842200 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.843702 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.845689 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.847141 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.847262 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.847483 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.848714 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.848892 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17788,"thread_start_us":17937,"threads_started":1}
I20260810 19:30:33.849058 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.822566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:43199 (local address 127.0.0.1:53806)
0810 19:30:33.840804 (+ 18238us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:33.840822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:33.841026 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:33.841646 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:33.841650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:33.841675 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:33.842132 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.842144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.844120 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.844126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.846268 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:33.846283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.848288 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.848294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:33.848458 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.848491 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:30:33.848584 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:30:33.848703 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18075,"mutex_wait_us":1,"thread_start_us":16177,"threads_started":1}
I20260810 19:30:33.849390 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:33.823193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.19.68.2:43199)
0810 19:30:33.838371 (+ 15178us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:33.838381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:33.840929 (+ 2548us) server_negotiation.cc:407] Connection header received
0810 19:30:33.841219 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:33.841237 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:33.841442 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:33.841652 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:33.842289 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.843961 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.846452 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:33.847824 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:33.847944 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:33.848114 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:33.848895 (+ 781us) server_negotiation.cc:299] Negotiation successful
0810 19:30:33.849059 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14983,"thread_start_us":15164,"threads_started":1}
I20260810 19:30:33.852759 17901 tablet_service.cc:1467] Processing CreateTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 (DEFAULT_TABLE table=test-table [id=8372ecbe90f74d1da14375af8f86c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:33.852788 17671 tablet_service.cc:1467] Processing CreateTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 (DEFAULT_TABLE table=test-table [id=8372ecbe90f74d1da14375af8f86c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:33.852789 18138 tablet_service.cc:1467] Processing CreateTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 (DEFAULT_TABLE table=test-table [id=8372ecbe90f74d1da14375af8f86c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:33.854000 17901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd3371f9abb46c1831d48cafdaa83a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:33.854171 17671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd3371f9abb46c1831d48cafdaa83a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:33.854197 18138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd3371f9abb46c1831d48cafdaa83a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:33.865149 18283 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Bootstrap starting.
I20260810 19:30:33.868544 18284 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Bootstrap starting.
I20260810 19:30:33.869320 18285 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Bootstrap starting.
I20260810 19:30:33.869566 18283 tablet_bootstrap.cc:654] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:33.870911 18283 log.cc:824] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:33.872495 18284 tablet_bootstrap.cc:654] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:33.872490 18285 tablet_bootstrap.cc:654] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:33.873474 18284 log.cc:824] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:33.873682 18285 log.cc:824] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:33.874184 18283 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: No bootstrap required, opened a new log
I20260810 19:30:33.874490 18283 ts_tablet_manager.cc:1397] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260810 19:30:33.876098 18285 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: No bootstrap required, opened a new log
I20260810 19:30:33.876328 18285 ts_tablet_manager.cc:1397] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 19:30:33.876394 18284 tablet_bootstrap.cc:492] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: No bootstrap required, opened a new log
I20260810 19:30:33.876643 18284 ts_tablet_manager.cc:1397] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:30:33.883579 18285 raft_consensus.cc:348] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.883980 18285 raft_consensus.cc:374] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:33.884127 18285 raft_consensus.cc:729] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5976931ac7a64a269557766adcd535aa, State: Initialized, Role: FOLLOWER
I20260810 19:30:33.884783 18285 consensus_queue.cc:260] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.885824 18283 raft_consensus.cc:348] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.886270 18283 raft_consensus.cc:374] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:33.886369 18283 raft_consensus.cc:729] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75b0491ca604f70b3992abc53a72078, State: Initialized, Role: FOLLOWER
I20260810 19:30:33.886935 18283 consensus_queue.cc:260] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.887001 18284 raft_consensus.cc:348] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.887444 18284 raft_consensus.cc:374] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:33.887584 18284 raft_consensus.cc:729] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b5f678f28f4b27b6da6029dc13d9e0, State: Initialized, Role: FOLLOWER
I20260810 19:30:33.888032 18285 ts_tablet_manager.cc:1428] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260810 19:30:33.888625 18284 consensus_queue.cc:260] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:33.889665 18253 heartbeater.cc:503] Master 127.19.68.62:34079 was elected leader, sending a full tablet report...
I20260810 19:30:33.890349 18284 ts_tablet_manager.cc:1428] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260810 19:30:33.896374 18019 heartbeater.cc:503] Master 127.19.68.62:34079 was elected leader, sending a full tablet report...
I20260810 19:30:33.897294 18283 ts_tablet_manager.cc:1428] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Time spent starting tablet: real 0.023s user 0.018s sys 0.001s
W20260810 19:30:33.945513 17783 tablet.cc:2369] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:30:33.975706 18020 tablet.cc:2369] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:30:33.998091 18255 tablet.cc:2369] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:34.079348 18291 raft_consensus.cc:482] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:34.079614 18291 raft_consensus.cc:504] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:34.081297 18291 leader_election.cc:288] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b5f678f28f4b27b6da6029dc13d9e0 (127.19.68.3:38581), b75b0491ca604f70b3992abc53a72078 (127.19.68.2:43199)
I20260810 19:30:34.088467 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.081714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:39545 (local address 127.19.68.3:38581)
0810 19:30:34.081905 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.081912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.082684 (+ 772us) server_negotiation.cc:407] Connection header received
0810 19:30:34.082882 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.082886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.082950 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.083025 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:34.083643 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.084606 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.086790 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.087550 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.087661 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.087760 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.088088 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.088210 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:30:34.088567 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.081848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:58717 (local address 127.19.68.2:43199)
0810 19:30:34.082042 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.082046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.083096 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 19:30:34.083329 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.083335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.083397 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.083491 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:34.084032 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.084919 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.086873 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.087635 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.087761 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.087855 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.088178 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.088318 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:30:34.088593 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.081782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38581 (local address 127.19.68.1:39545)
0810 19:30:34.082570 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:34.082587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:34.082739 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:34.083060 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:34.083065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:34.083092 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:34.083498 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:34.083511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.084739 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.084745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:34.086632 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:34.086644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.087747 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.087754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:34.087936 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.087961 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:30:34.088040 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:30:34.088161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":20,"thread_start_us":458,"threads_started":1}
I20260810 19:30:34.088663 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.082059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:43199 (local address 127.19.68.1:58717)
0810 19:30:34.082986 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:34.083005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:34.083140 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:34.083508 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:34.083514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:34.083537 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:34.083863 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:34.083872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.085111 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.085121 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:34.086704 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:34.086717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.087880 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.087888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:34.088029 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.088052 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:30:34.088128 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:30:34.088244 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":19,"thread_start_us":577,"threads_started":1}
I20260810 19:30:34.089790 17932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd3371f9abb46c1831d48cafdaa83a2" candidate_uuid: "5976931ac7a64a269557766adcd535aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75b0491ca604f70b3992abc53a72078" is_pre_election: true
I20260810 19:30:34.090284 17932 raft_consensus.cc:2454] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5976931ac7a64a269557766adcd535aa in term 0.
I20260810 19:30:34.090971 17601 leader_election.cc:302] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5976931ac7a64a269557766adcd535aa, b75b0491ca604f70b3992abc53a72078; no voters:
I20260810 19:30:34.091101 18177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd3371f9abb46c1831d48cafdaa83a2" candidate_uuid: "5976931ac7a64a269557766adcd535aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" is_pre_election: true
I20260810 19:30:34.091403 18291 raft_consensus.cc:2781] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:34.091540 18291 raft_consensus.cc:482] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:34.091639 18291 raft_consensus.cc:3037] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:34.091615 18177 raft_consensus.cc:2454] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5976931ac7a64a269557766adcd535aa in term 0.
I20260810 19:30:34.094027 18291 raft_consensus.cc:504] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:34.094569 18291 leader_election.cc:288] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [CANDIDATE]: Term 1 election: Requested vote from peers 28b5f678f28f4b27b6da6029dc13d9e0 (127.19.68.3:38581), b75b0491ca604f70b3992abc53a72078 (127.19.68.2:43199)
I20260810 19:30:34.095012 18177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd3371f9abb46c1831d48cafdaa83a2" candidate_uuid: "5976931ac7a64a269557766adcd535aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b5f678f28f4b27b6da6029dc13d9e0"
I20260810 19:30:34.095013 17932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd3371f9abb46c1831d48cafdaa83a2" candidate_uuid: "5976931ac7a64a269557766adcd535aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75b0491ca604f70b3992abc53a72078"
I20260810 19:30:34.095186 18177 raft_consensus.cc:3037] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:34.095183 17932 raft_consensus.cc:3037] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:34.097363 18177 raft_consensus.cc:2454] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5976931ac7a64a269557766adcd535aa in term 1.
I20260810 19:30:34.097397 17932 raft_consensus.cc:2454] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5976931ac7a64a269557766adcd535aa in term 1.
I20260810 19:30:34.097736 17600 leader_election.cc:302] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b5f678f28f4b27b6da6029dc13d9e0, 5976931ac7a64a269557766adcd535aa; no voters:
I20260810 19:30:34.098029 18291 raft_consensus.cc:2781] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:34.098474 18291 raft_consensus.cc:686] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 1 LEADER]: Becoming Leader. State: Replica: 5976931ac7a64a269557766adcd535aa, State: Running, Role: LEADER
I20260810 19:30:34.098925 18291 consensus_queue.cc:237] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } }
I20260810 19:30:34.103528 17450 catalog_manager.cc:5495] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5976931ac7a64a269557766adcd535aa (127.19.68.1). New cstate: current_term: 1 leader_uuid: "5976931ac7a64a269557766adcd535aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5976931ac7a64a269557766adcd535aa" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43493 } health_report { overall_health: HEALTHY } } }
I20260810 19:30:34.143392 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.138067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.19.68.1:43493)
0810 19:30:34.138270 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.138276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.138352 (+ 76us) server_negotiation.cc:407] Connection header received
0810 19:30:34.138592 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.138597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.138687 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.138819 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:34.139215 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.139939 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.141318 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.141724 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.141835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.142338 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:34.142826 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:34.142984 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.143061 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.143141 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:30:34.151417 17932 raft_consensus.cc:1261] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 1 FOLLOWER]: Refusing update from remote peer 5976931ac7a64a269557766adcd535aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:34.151417 18177 raft_consensus.cc:1261] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 1 FOLLOWER]: Refusing update from remote peer 5976931ac7a64a269557766adcd535aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:34.152554 18291 consensus_queue.cc:1035] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b5f678f28f4b27b6da6029dc13d9e0" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:34.152882 18298 consensus_queue.cc:1035] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75b0491ca604f70b3992abc53a72078" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:34.163832 18303 mvcc.cc:205] Tried to move back new op lower bound from 7317054399074336768 to 7317054398872563712. Current Snapshot: MvccSnapshot[applied={T|T < 7317054399074336768}]
I20260810 19:30:34.176890 17449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:34.177201 17449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:34.180745 17449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 161afe1821ad4bd3b79151885c4fc44a: Sending DeleteTablet for 3 replicas of tablet afd3371f9abb46c1831d48cafdaa83a2
I20260810 19:30:34.182019 18138 tablet_service.cc:1514] Processing DeleteTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:34 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:51454
I20260810 19:30:34.182255 17671 tablet_service.cc:1514] Processing DeleteTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:34 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:52118
I20260810 19:30:34.182828 18316 tablet_replica.cc:330] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: stopping tablet replica
I20260810 19:30:34.182756 17901 tablet_service.cc:1514] Processing DeleteTablet for tablet afd3371f9abb46c1831d48cafdaa83a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:34 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:53806
I20260810 19:30:34.183219 18316 raft_consensus.cc:2229] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:34.183749 18316 raft_consensus.cc:2258] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:34.184885 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 17553
I20260810 19:30:34.190369 18316 ts_tablet_manager.cc:1905] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:34.195464 18319 tablet_replica.cc:330] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: stopping tablet replica
I20260810 19:30:34.196056 18319 raft_consensus.cc:2229] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:34.196710 18319 raft_consensus.cc:2258] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:34.199025 18319 ts_tablet_manager.cc:1905] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:34.205747 18316 ts_tablet_manager.cc:1918] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:34.206264 18319 ts_tablet_manager.cc:1918] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:34.206858 18316 log.cc:1196] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/afd3371f9abb46c1831d48cafdaa83a2
I20260810 19:30:34.207023 18317 tablet_replica.cc:330] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: stopping tablet replica
I20260810 19:30:34.207507 18316 ts_tablet_manager.cc:1939] T afd3371f9abb46c1831d48cafdaa83a2 P 28b5f678f28f4b27b6da6029dc13d9e0: Deleting consensus metadata
I20260810 19:30:34.207576 18317 raft_consensus.cc:2229] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 1 LEADER]: Raft consensus shutting down.
Aug 10 19:30:34 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:30:34.208415 18317 raft_consensus.cc:2258] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:34.208873 18319 log.cc:1196] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/afd3371f9abb46c1831d48cafdaa83a2
I20260810 19:30:34.209885 17425 catalog_manager.cc:4841] TS 28b5f678f28f4b27b6da6029dc13d9e0 (127.19.68.3:38581): tablet afd3371f9abb46c1831d48cafdaa83a2 (table test-table [id=8372ecbe90f74d1da14375af8f86c392]) successfully deleted
I20260810 19:30:34.210889 18317 ts_tablet_manager.cc:1905] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:34.214398 18319 ts_tablet_manager.cc:1939] T afd3371f9abb46c1831d48cafdaa83a2 P b75b0491ca604f70b3992abc53a72078: Deleting consensus metadata
I20260810 19:30:34.216451 17426 catalog_manager.cc:4841] TS b75b0491ca604f70b3992abc53a72078 (127.19.68.2:43199): tablet afd3371f9abb46c1831d48cafdaa83a2 (table test-table [id=8372ecbe90f74d1da14375af8f86c392]) successfully deleted
I20260810 19:30:34.217602 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.185831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.19.68.1:43493)
0810 19:30:34.186025 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.186031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.197321 (+ 11290us) server_negotiation.cc:407] Connection header received
0810 19:30:34.197508 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.197513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.197600 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.197721 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:34.199554 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.200813 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.202890 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.203631 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.203746 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.209471 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:34.209487 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:34.209504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:34.209561 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:34.213889 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.214807 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.214817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.214821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.215037 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.215542 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.215547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.215551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.216063 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:34.216348 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.216944 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.217229 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:30:34.217676 18317 ts_tablet_manager.cc:1918] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:34.217823 18317 log.cc:1196] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/afd3371f9abb46c1831d48cafdaa83a2
I20260810 19:30:34.218472 18317 ts_tablet_manager.cc:1939] T afd3371f9abb46c1831d48cafdaa83a2 P 5976931ac7a64a269557766adcd535aa: Deleting consensus metadata
I20260810 19:30:34.219270 17776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:34.325666 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 17553
I20260810 19:30:34.328161 17424 catalog_manager.cc:4841] TS 5976931ac7a64a269557766adcd535aa (127.19.68.1:43493): tablet afd3371f9abb46c1831d48cafdaa83a2 (table test-table [id=8372ecbe90f74d1da14375af8f86c392]) successfully deleted
I20260810 19:30:34.347982 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 17790
Aug 10 19:30:34 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:30:34.362545 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.348895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.19.68.2:43199)
0810 19:30:34.349078 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.349106 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.349132 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:30:34.349204 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.349208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.349294 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.349407 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:34.350784 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.351560 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.352786 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.353237 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.353338 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.357455 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:34.357471 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:34.357486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:34.357519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:34.360258 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.360859 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.360866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.360868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.360983 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.361312 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.361316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.361318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.361694 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:34.361928 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.362100 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.362271 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:30:34.363246 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:34.452647 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 17790
I20260810 19:30:34.481058 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 18032
Aug 10 19:30:34 dist-test-slave-2j7r krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:30:34.497023 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:34.482033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.19.68.3:38581)
0810 19:30:34.482233 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:34.482240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:34.482263 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:30:34.482492 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:34.482497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:34.482571 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:34.482688 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:34.484156 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.485070 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.486584 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:34.487007 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:34.487134 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:34.491606 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:34.491621 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:34.491638 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:34.491673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:34.494273 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.494967 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.494975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.494977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.495127 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:34.495503 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:34.495507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:34.495509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:34.495861 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:34.496151 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:34.496462 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 19:30:34.496673 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:30:34.497761 18243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:34.584767 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 18032
I20260810 19:30:34.604830 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 17394
I20260810 19:30:34.606065 17504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:34.715185 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 17394
2026-08-10T19:30:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4740 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T19:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:34Z Disabled control of system clock
I20260810 19:30:34.814697 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37875
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:35.141746 18342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:35.142026 18342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:35.142113 18342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:35.149109 18342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:35.149196 18342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:35.149240 18342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:35.149293 18342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:35.149336 18342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:35.157989 18342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:37875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:35.159217 18342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:35.160895 18342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:35.171913 18353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:35.171877 18357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:35.172130 18350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:35.172816 18342 server_base.cc:1034] running on GCE node
I20260810 19:30:35.173563 18342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:35.174880 18342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:35.176090 18342 hybrid_clock.cc:648] HybridClock initialized: now 1786390235176047 us; error 35 us; skew 500 ppm
I20260810 19:30:35.176433 18342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:35.178628 18342 webserver.cc:460] Webserver started at http://127.19.68.62:35155/ using document root <none> and password file <none>
I20260810 19:30:35.179275 18342 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:35.179365 18342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:35.179625 18342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:35.182426 18342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "e7c6f16822b94f2d9a9403b0c24b1650"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "c03a230b0bc61e1c59823c7af7771085"
server_key_iv: "bf3dc2c952301e8911d22c6200f9376c"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.183156 18342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "e7c6f16822b94f2d9a9403b0c24b1650"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "c03a230b0bc61e1c59823c7af7771085"
server_key_iv: "bf3dc2c952301e8911d22c6200f9376c"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.187932 18342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:30:35.191340 18367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:35.192574 18342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:30:35.192736 18342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "e7c6f16822b94f2d9a9403b0c24b1650"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "c03a230b0bc61e1c59823c7af7771085"
server_key_iv: "bf3dc2c952301e8911d22c6200f9376c"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.192951 18342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:35.203935 18342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:35.204763 18342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:35.205050 18342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:35.223217 18342 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:37875
I20260810 19:30:35.223245 18449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:37875 every 8 connection(s)
I20260810 19:30:35.224545 18342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:35.228420 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:35.229859 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 18342
I20260810 19:30:35.230002 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:35.230487 19728 external_mini_cluster.cc:1428] Setting key ea10092121ec343673a81650dd5d3aaf
I20260810 19:30:35.241963 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: Bootstrap starting.
I20260810 19:30:35.244128 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:35.232287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.19.68.62:37875)
0810 19:30:35.233350 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:35.233362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:35.233410 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:30:35.234962 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:35.234983 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:35.235428 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:35.235728 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:35.236377 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.237778 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.239679 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.240361 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.240490 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:35.241839 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:35.241848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:35.241860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:35.242420 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:35.242503 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:35.242525 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:35.242617 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:35.242865 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 19:30:35.243171 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":381,"threads_started":1}
I20260810 19:30:35.245225 18452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:35.246553 18452 log.cc:824] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:35.249119 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: No bootstrap required, opened a new log
I20260810 19:30:35.255997 18452 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } }
I20260810 19:30:35.256315 18452 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:35.256386 18452 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c6f16822b94f2d9a9403b0c24b1650, State: Initialized, Role: FOLLOWER
I20260810 19:30:35.256839 18452 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } }
I20260810 19:30:35.257042 18452 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:35.257164 18452 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:35.257304 18452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:35.259500 18452 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } }
I20260810 19:30:35.259929 18452 leader_election.cc:302] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c6f16822b94f2d9a9403b0c24b1650; no voters:
I20260810 19:30:35.260448 18452 leader_election.cc:288] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:35.260584 18461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:35.261340 18461 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [term 1 LEADER]: Becoming Leader. State: Replica: e7c6f16822b94f2d9a9403b0c24b1650, State: Running, Role: LEADER
I20260810 19:30:35.261817 18461 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } }
I20260810 19:30:35.262502 18452 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:35.264607 18465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c6f16822b94f2d9a9403b0c24b1650. Latest consensus state: current_term: 1 leader_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } } }
I20260810 19:30:35.264632 18464 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c6f16822b94f2d9a9403b0c24b1650" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 37875 } } }
I20260810 19:30:35.265108 18464 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:35.265105 18465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:35.266810 18470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:35.272470 18470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:35.281002 18470 catalog_manager.cc:1347] Generated new cluster ID: 0e273baa47f348578fb51ef10d04661a
I20260810 19:30:35.281128 18470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:35.328215 18470 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:35.329546 18470 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:35.338907 18470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: Generated new TSK 0
I20260810 19:30:35.339684 18470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:35.354483 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--builtin_ntp_servers=127.19.68.20:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:35.689983 18494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:35.690258 18494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:35.690392 18494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:35.697616 18494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:35.697711 18494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:35.697857 18494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:35.706956 18494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:35.708199 18494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:35.710017 18494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:35.722322 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:35.722458 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:35.722430 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:35.723369 18494 server_base.cc:1034] running on GCE node
I20260810 19:30:35.723960 18494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:35.724821 18494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:35.726017 18494 hybrid_clock.cc:648] HybridClock initialized: now 1786390235725970 us; error 36 us; skew 500 ppm
I20260810 19:30:35.726308 18494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:35.728433 18494 webserver.cc:460] Webserver started at http://127.19.68.1:36513/ using document root <none> and password file <none>
I20260810 19:30:35.729066 18494 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:35.729204 18494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:35.729528 18494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:35.732250 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "c37e486805f94d8dbc777fb726d17934"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "d949c7fc621cf70749b0138bc11188eb"
server_key_iv: "79feb1d9c80e0090e8764f6a347874fc"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.733016 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "c37e486805f94d8dbc777fb726d17934"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "d949c7fc621cf70749b0138bc11188eb"
server_key_iv: "79feb1d9c80e0090e8764f6a347874fc"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.738023 18494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 19:30:35.741324 18515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:35.742653 18494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:30:35.742828 18494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "c37e486805f94d8dbc777fb726d17934"
format_stamp: "Formatted at 2026-08-10 19:30:35 on dist-test-slave-2j7r"
server_key: "d949c7fc621cf70749b0138bc11188eb"
server_key_iv: "79feb1d9c80e0090e8764f6a347874fc"
server_key_version: "encryptionkey@0"
I20260810 19:30:35.743047 18494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:35.773370 18494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:35.774279 18494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:35.774571 18494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:35.775660 18494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:35.777355 18494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:35.777463 18494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:35.777590 18494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:35.777675 18494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:35.810148 18494 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:33791
I20260810 19:30:35.810213 18717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:33791 every 8 connection(s)
I20260810 19:30:35.811452 18494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:35.815017 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 18494
I20260810 19:30:35.815196 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:35.815795 19728 external_mini_cluster.cc:1428] Setting key f363edd64836dd2d639a39a1eb3ba2c1
I20260810 19:30:35.819021 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--builtin_ntp_servers=127.19.68.20:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:35.824853 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:35.813911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:56589 (local address 127.19.68.62:37875)
0810 19:30:35.814365 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:35.814369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:35.815661 (+ 1292us) server_negotiation.cc:407] Connection header received
0810 19:30:35.817800 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:35.817805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:35.817885 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:35.817978 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:35.819382 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.820117 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.822199 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.822612 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.822706 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:35.824165 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:35.824170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:35.824173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:35.824190 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:35.824228 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:35.824240 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:35.824314 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:35.824515 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 19:30:35.824591 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260810 19:30:35.826195 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:35.814362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37875 (local address 127.19.68.1:56589)
0810 19:30:35.815476 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:35.815539 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:35.817492 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:35.818289 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:35.818301 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:35.818412 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:35.819195 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:35.819239 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.820268 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.820273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:35.822062 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:35.822077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:35.822825 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:35.822831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:35.823095 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:35.823836 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:35.823839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:35.823888 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:35.823978 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:35.824325 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:35.824336 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:35.824342 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:30:35.824503 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 19:30:35.824905 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":303,"threads_started":1}
I20260810 19:30:35.827731 18719 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37875
I20260810 19:30:35.827996 18719 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:35.828714 18719 heartbeater.cc:511] Master 127.19.68.62:37875 requested a full tablet report, sending...
I20260810 19:30:35.830847 18395 ts_manager.cc:194] Registered new tserver with Master: c37e486805f94d8dbc777fb726d17934 (127.19.68.1:33791)
I20260810 19:30:35.832696 18395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.1:56589
W20260810 19:30:36.150341 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:36.150616 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:36.150741 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:36.157893 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:36.157990 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:36.158120 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:36.167074 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:36.168391 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:36.170235 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:36.182773 18740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:36.182849 18743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:36.182821 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:36.183787 18725 server_base.cc:1034] running on GCE node
I20260810 19:30:36.184365 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:36.185278 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:36.186486 18725 hybrid_clock.cc:648] HybridClock initialized: now 1786390236186439 us; error 35 us; skew 500 ppm
I20260810 19:30:36.186779 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:36.188869 18725 webserver.cc:460] Webserver started at http://127.19.68.2:40715/ using document root <none> and password file <none>
I20260810 19:30:36.189548 18725 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:36.189649 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:36.189954 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:36.192711 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "135e99eb8ff14cf09eaa5fa3ad449430"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "5e633e01676381a9bb101bf7e4d2e93e"
server_key_iv: "03e474a941035901bc74eabc8a07dc6f"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.193498 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "135e99eb8ff14cf09eaa5fa3ad449430"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "5e633e01676381a9bb101bf7e4d2e93e"
server_key_iv: "03e474a941035901bc74eabc8a07dc6f"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.198526 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:30:36.202013 18753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.203389 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:30:36.203562 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "135e99eb8ff14cf09eaa5fa3ad449430"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "5e633e01676381a9bb101bf7e4d2e93e"
server_key_iv: "03e474a941035901bc74eabc8a07dc6f"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.203764 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:36.216939 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:36.217792 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:36.218063 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:36.219027 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:36.220417 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:36.220515 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.220633 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:36.220717 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.252292 18725 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:39611
I20260810 19:30:36.252362 18950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:39611 every 8 connection(s)
I20260810 19:30:36.253604 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:36.261677 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 18725
I20260810 19:30:36.261844 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:36.262275 19728 external_mini_cluster.cc:1428] Setting key 7449142b4d49ab83913a31ddcef8c314
I20260810 19:30:36.265280 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--builtin_ntp_servers=127.19.68.20:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:36.265995 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.256047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:35409 (local address 127.19.68.62:37875)
0810 19:30:36.256228 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.256233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.257573 (+ 1340us) server_negotiation.cc:407] Connection header received
0810 19:30:36.259350 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.259356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.259438 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.259529 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:36.260614 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.261457 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.263217 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.263642 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.263739 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.265127 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:36.265135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:36.265140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:36.265163 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:36.265202 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:36.265220 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:36.265335 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.265551 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.265651 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:30:36.266958 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.256528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37875 (local address 127.19.68.2:35409)
0810 19:30:36.257427 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.257464 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.259071 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.259782 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.259790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.259876 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:36.260445 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.260463 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.261614 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.261618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.263088 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.263101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.263872 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.263880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.264135 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.264784 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:36.264786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:36.264822 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:36.264893 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:36.265324 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:36.265332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:36.265337 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.265530 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.265902 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":261,"threads_started":1}
I20260810 19:30:36.268422 18952 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37875
I20260810 19:30:36.268728 18952 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:36.269640 18952 heartbeater.cc:511] Master 127.19.68.62:37875 requested a full tablet report, sending...
I20260810 19:30:36.271061 18395 ts_manager.cc:194] Registered new tserver with Master: 135e99eb8ff14cf09eaa5fa3ad449430 (127.19.68.2:39611)
I20260810 19:30:36.271826 18395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.2:35409
W20260810 19:30:36.599458 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:36.599716 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:36.599812 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:36.606844 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:36.606935 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:36.607038 18958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:36.615726 18958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:36.616904 18958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:36.618614 18958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:36.631134 18966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:36.631140 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:36.631079 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:36.631922 18958 server_base.cc:1034] running on GCE node
I20260810 19:30:36.632467 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:36.633458 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:36.634662 18958 hybrid_clock.cc:648] HybridClock initialized: now 1786390236634613 us; error 44 us; skew 500 ppm
I20260810 19:30:36.634940 18958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:36.637106 18958 webserver.cc:460] Webserver started at http://127.19.68.3:40135/ using document root <none> and password file <none>
I20260810 19:30:36.637778 18958 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:36.637887 18958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:36.638187 18958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:36.640969 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "5cbee539c7244c96aca1bcc7d09784b4"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "ad8d6cff7a799feac55d2f63b2dcb92a"
server_key_iv: "72d779a28f5e0a89d1465cc9593e68ce"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.641783 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "5cbee539c7244c96aca1bcc7d09784b4"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "ad8d6cff7a799feac55d2f63b2dcb92a"
server_key_iv: "72d779a28f5e0a89d1465cc9593e68ce"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.646632 18958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:30:36.649957 18983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.651124 18958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 19:30:36.651263 18958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "5cbee539c7244c96aca1bcc7d09784b4"
format_stamp: "Formatted at 2026-08-10 19:30:36 on dist-test-slave-2j7r"
server_key: "ad8d6cff7a799feac55d2f63b2dcb92a"
server_key_iv: "72d779a28f5e0a89d1465cc9593e68ce"
server_key_version: "encryptionkey@0"
I20260810 19:30:36.651456 18958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:36.670276 18958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:36.671003 18958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:36.671255 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:36.672166 18958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:36.673547 18958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:36.673641 18958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.673745 18958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:36.673816 18958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:36.703879 18958 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:43339
I20260810 19:30:36.703951 19166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:43339 every 8 connection(s)
I20260810 19:30:36.705132 18958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:36.706312 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 18958
I20260810 19:30:36.706502 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:36.707094 19728 external_mini_cluster.cc:1428] Setting key 87a746d55053b5c0ef77054998f69300
I20260810 19:30:36.717528 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.707795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:55529 (local address 127.19.68.62:37875)
0810 19:30:36.707990 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.707996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.709579 (+ 1583us) server_negotiation.cc:407] Connection header received
0810 19:30:36.711410 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.711415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.711499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.711586 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:36.712610 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.713458 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.714964 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.715398 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.715508 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.716719 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:36.716723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:36.716727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:36.716743 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:36.716778 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:36.716790 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:36.716864 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.717075 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.717200 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:30:36.718441 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.708457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:37875 (local address 127.19.68.3:55529)
0810 19:30:36.709440 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.709477 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.711160 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.711828 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.711835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.711916 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:36.712450 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.712466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.713598 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.713601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.714825 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.714837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.715608 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.715616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.715860 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.716432 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:36.716434 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:36.716469 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:36.716541 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:36.716880 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:36.716891 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:36.716898 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.717046 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.717423 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260810 19:30:36.719678 19171 heartbeater.cc:347] Connected to a master server at 127.19.68.62:37875
I20260810 19:30:36.719947 19171 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:36.720701 19171 heartbeater.cc:511] Master 127.19.68.62:37875 requested a full tablet report, sending...
I20260810 19:30:36.722301 18394 ts_manager.cc:194] Registered new tserver with Master: 5cbee539c7244c96aca1bcc7d09784b4 (127.19.68.3:43339)
I20260810 19:30:36.723150 18394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.3:55529
I20260810 19:30:36.734390 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:36.743661 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.738342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.19.68.62:37875)
0810 19:30:36.738539 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.738544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.738944 (+ 400us) server_negotiation.cc:407] Connection header received
0810 19:30:36.739074 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.739077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.739144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.739223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:36.739621 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.740275 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.741633 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.742031 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.742121 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.743126 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:36.743131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:36.743134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:36.743149 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:36.743179 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:36.743190 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:36.743254 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.743355 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.743423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:30:36.747335 18394 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:36.754099 18394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:36.794564 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.19.68.2:39611)
0810 19:30:36.785582 (+ 18762us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.785592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.786152 (+ 560us) server_negotiation.cc:407] Connection header received
0810 19:30:36.786461 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.786469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.786730 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.786907 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:36.787649 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.789150 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.791766 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.793233 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.793353 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.793592 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.794099 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.794259 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18592,"thread_start_us":18467,"threads_started":1}
I20260810 19:30:36.794535 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.766195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:39611 (local address 127.0.0.1:54940)
0810 19:30:36.786041 (+ 19846us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.786059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.786261 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.787151 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.787157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.787180 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:36.787473 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.787481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.789312 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.789317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.791556 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.791572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.793732 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.793738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.793881 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.793914 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.794021 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.794201 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19699,"thread_start_us":19549,"threads_started":1}
I20260810 19:30:36.795490 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.765899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:43339 (local address 127.0.0.1:36802)
0810 19:30:36.780522 (+ 14623us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.780570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.780777 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.788624 (+ 7847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.788628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.788651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:36.789042 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.789056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.791018 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.791023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.792884 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.792898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.794748 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.794754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.794904 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.794949 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.795017 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.795134 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14321,"mutex_wait_us":85,"thread_start_us":22210,"threads_started":1}
I20260810 19:30:36.796329 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.766855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.19.68.1:33791)
0810 19:30:36.779872 (+ 13017us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.779883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.788471 (+ 8588us) server_negotiation.cc:407] Connection header received
0810 19:30:36.788620 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.788627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.788832 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.789033 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:36.789995 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.791310 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.793761 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.795150 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.795277 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.795510 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.795953 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.796080 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12796,"thread_start_us":14072,"threads_started":1}
I20260810 19:30:36.796640 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.766125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.19.68.3:43339)
0810 19:30:36.788082 (+ 21957us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.788090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.788124 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:30:36.788214 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.788222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.788420 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.788641 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:36.789270 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.790869 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.793145 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.794521 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.795743 (+ 1222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.796015 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.796078 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.796221 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21779,"thread_start_us":21623,"threads_started":1}
I20260810 19:30:36.796700 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.766547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60290)
0810 19:30:36.788344 (+ 21797us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.788358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.788479 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.789315 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.789319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.789345 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:36.789761 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.789775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.791446 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.791454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.793556 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.793571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.795350 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.795354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.795498 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.795529 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.795611 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.795733 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21692,"thread_start_us":13515,"threads_started":1}
I20260810 19:30:36.799633 18848 tablet_service.cc:1467] Processing CreateTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 (DEFAULT_TABLE table=test-table [id=1240f0d8e8194be8b9b9dcaac5f0e5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:36.800215 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 (DEFAULT_TABLE table=test-table [id=1240f0d8e8194be8b9b9dcaac5f0e5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:36.800803 18848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38698a7b0cb249789d3f7f6e992b4015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:36.800995 19057 tablet_service.cc:1467] Processing CreateTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 (DEFAULT_TABLE table=test-table [id=1240f0d8e8194be8b9b9dcaac5f0e5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:36.801630 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38698a7b0cb249789d3f7f6e992b4015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:36.802385 19057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38698a7b0cb249789d3f7f6e992b4015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:36.821645 19200 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Bootstrap starting.
I20260810 19:30:36.822376 19203 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Bootstrap starting.
I20260810 19:30:36.825501 19200 tablet_bootstrap.cc:654] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:36.826714 19203 tablet_bootstrap.cc:654] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:36.826908 19200 log.cc:824] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:36.827533 19204 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Bootstrap starting.
I20260810 19:30:36.829444 19203 log.cc:824] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:36.830438 19200 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: No bootstrap required, opened a new log
I20260810 19:30:36.830720 19200 ts_tablet_manager.cc:1397] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:30:36.831105 19204 tablet_bootstrap.cc:654] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:36.831969 19203 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: No bootstrap required, opened a new log
I20260810 19:30:36.832197 19203 ts_tablet_manager.cc:1397] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 19:30:36.832300 19204 log.cc:824] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:36.835292 19204 tablet_bootstrap.cc:492] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: No bootstrap required, opened a new log
I20260810 19:30:36.835567 19204 ts_tablet_manager.cc:1397] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:30:36.836036 18719 heartbeater.cc:503] Master 127.19.68.62:37875 was elected leader, sending a full tablet report...
I20260810 19:30:36.840544 19200 raft_consensus.cc:348] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.841301 19200 raft_consensus.cc:374] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:36.841007 19203 raft_consensus.cc:348] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.841457 19203 raft_consensus.cc:374] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:36.841490 19200 raft_consensus.cc:729] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135e99eb8ff14cf09eaa5fa3ad449430, State: Initialized, Role: FOLLOWER
I20260810 19:30:36.841617 19203 raft_consensus.cc:729] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbee539c7244c96aca1bcc7d09784b4, State: Initialized, Role: FOLLOWER
I20260810 19:30:36.842185 19203 consensus_queue.cc:260] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.842160 19200 consensus_queue.cc:260] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.843185 19171 heartbeater.cc:503] Master 127.19.68.62:37875 was elected leader, sending a full tablet report...
I20260810 19:30:36.843912 19203 ts_tablet_manager.cc:1428] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 19:30:36.846359 19200 ts_tablet_manager.cc:1428] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 19:30:36.846513 18952 heartbeater.cc:503] Master 127.19.68.62:37875 was elected leader, sending a full tablet report...
I20260810 19:30:36.848009 19204 raft_consensus.cc:348] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.848373 19204 raft_consensus.cc:374] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:36.848470 19204 raft_consensus.cc:729] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e486805f94d8dbc777fb726d17934, State: Initialized, Role: FOLLOWER
I20260810 19:30:36.849030 19204 consensus_queue.cc:260] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.850718 19204 ts_tablet_manager.cc:1428] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260810 19:30:36.926834 19216 raft_consensus.cc:482] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:36.927181 19216 raft_consensus.cc:504] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.928925 19216 leader_election.cc:288] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbee539c7244c96aca1bcc7d09784b4 (127.19.68.3:43339), c37e486805f94d8dbc777fb726d17934 (127.19.68.1:33791)
I20260810 19:30:36.935482 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.929555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:47431 (local address 127.19.68.3:43339)
0810 19:30:36.929770 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.929775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.930443 (+ 668us) server_negotiation.cc:407] Connection header received
0810 19:30:36.930649 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.930656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.930724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.930810 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:36.931333 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.932045 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.933869 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.934635 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.934752 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.934871 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.935115 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.935242 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:30:36.935539 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.929407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:43339 (local address 127.19.68.2:47431)
0810 19:30:36.930329 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.930345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.930509 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.930850 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.930855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.930893 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:36.931199 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.931208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.932178 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.932183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.933682 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.933695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.934824 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.934830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.934969 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.934993 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.935074 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.935202 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":406,"threads_started":1}
I20260810 19:30:36.935887 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.929399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49139 (local address 127.19.68.1:33791)
0810 19:30:36.929592 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.929597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.930928 (+ 1331us) server_negotiation.cc:407] Connection header received
0810 19:30:36.931090 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.931096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.931170 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.931246 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:36.931783 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.932467 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.934203 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.934969 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.935086 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.935174 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.935486 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.935616 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:30:36.935875 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.929824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.19.68.2:49139)
0810 19:30:36.930814 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:36.930829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:36.930966 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:36.931286 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:36.931290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:36.931313 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:36.931622 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.931630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.932613 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.932618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.934023 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:36.934036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.935151 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.935157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:36.935338 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.935362 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:30:36.935440 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 19:30:36.935566 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":657,"threads_started":1}
I20260810 19:30:36.936730 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38698a7b0cb249789d3f7f6e992b4015" candidate_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbee539c7244c96aca1bcc7d09784b4" is_pre_election: true
I20260810 19:30:36.936844 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38698a7b0cb249789d3f7f6e992b4015" candidate_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e486805f94d8dbc777fb726d17934" is_pre_election: true
I20260810 19:30:36.937335 19090 raft_consensus.cc:2454] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135e99eb8ff14cf09eaa5fa3ad449430 in term 0.
I20260810 19:30:36.937383 18635 raft_consensus.cc:2454] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135e99eb8ff14cf09eaa5fa3ad449430 in term 0.
I20260810 19:30:36.937963 18760 leader_election.cc:302] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135e99eb8ff14cf09eaa5fa3ad449430, 5cbee539c7244c96aca1bcc7d09784b4; no voters:
I20260810 19:30:36.938315 19216 raft_consensus.cc:2781] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:36.938441 19216 raft_consensus.cc:482] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:36.938532 19216 raft_consensus.cc:3037] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:36.941136 19216 raft_consensus.cc:504] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.941704 19216 leader_election.cc:288] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [CANDIDATE]: Term 1 election: Requested vote from peers 5cbee539c7244c96aca1bcc7d09784b4 (127.19.68.3:43339), c37e486805f94d8dbc777fb726d17934 (127.19.68.1:33791)
I20260810 19:30:36.942164 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38698a7b0cb249789d3f7f6e992b4015" candidate_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbee539c7244c96aca1bcc7d09784b4"
I20260810 19:30:36.942176 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38698a7b0cb249789d3f7f6e992b4015" candidate_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e486805f94d8dbc777fb726d17934"
I20260810 19:30:36.942344 18635 raft_consensus.cc:3037] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:36.942341 19090 raft_consensus.cc:3037] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:36.944576 19090 raft_consensus.cc:2454] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135e99eb8ff14cf09eaa5fa3ad449430 in term 1.
I20260810 19:30:36.944577 18635 raft_consensus.cc:2454] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135e99eb8ff14cf09eaa5fa3ad449430 in term 1.
I20260810 19:30:36.944965 18760 leader_election.cc:302] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135e99eb8ff14cf09eaa5fa3ad449430, 5cbee539c7244c96aca1bcc7d09784b4; no voters:
I20260810 19:30:36.945269 19216 raft_consensus.cc:2781] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:36.945722 19216 raft_consensus.cc:686] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 1 LEADER]: Becoming Leader. State: Replica: 135e99eb8ff14cf09eaa5fa3ad449430, State: Running, Role: LEADER
I20260810 19:30:36.946208 19216 consensus_queue.cc:237] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } }
I20260810 19:30:36.951176 18395 catalog_manager.cc:5495] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 135e99eb8ff14cf09eaa5fa3ad449430 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "135e99eb8ff14cf09eaa5fa3ad449430" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 } health_report { overall_health: UNKNOWN } } }
W20260810 19:30:36.956727 19172 tablet.cc:2369] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:36.988933 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:36.983426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.19.68.2:39611)
0810 19:30:36.983645 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:36.983651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:36.983671 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:30:36.983940 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:36.983944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:36.984028 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:36.984166 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:36.984560 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.985288 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.986726 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:36.987177 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:36.987298 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:36.987815 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:36.988336 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:36.988471 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:36.988580 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 19:30:36.988666 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:30:36.997071 19090 raft_consensus.cc:1261] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 1 FOLLOWER]: Refusing update from remote peer 135e99eb8ff14cf09eaa5fa3ad449430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:36.997066 18635 raft_consensus.cc:1261] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 1 FOLLOWER]: Refusing update from remote peer 135e99eb8ff14cf09eaa5fa3ad449430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:36.998044 19220 consensus_queue.cc:1035] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e486805f94d8dbc777fb726d17934" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:36.998414 19216 consensus_queue.cc:1035] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbee539c7244c96aca1bcc7d09784b4" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:30:37.005654 18953 tablet.cc:2369] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:37.013688 19223 mvcc.cc:205] Tried to move back new op lower bound from 7317054410729734144 to 7317054410535985152. Current Snapshot: MvccSnapshot[applied={T|T < 7317054410729734144}]
I20260810 19:30:37.024945 18395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:37.025230 18395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:37.028545 18395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7c6f16822b94f2d9a9403b0c24b1650: Sending DeleteTablet for 3 replicas of tablet 38698a7b0cb249789d3f7f6e992b4015
I20260810 19:30:37.029613 19057 tablet_service.cc:1514] Processing DeleteTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:37 UTC) from {username='slave'} at 127.0.0.1:36802
I20260810 19:30:37.029639 18848 tablet_service.cc:1514] Processing DeleteTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:37 UTC) from {username='slave'} at 127.0.0.1:54940
I20260810 19:30:37.029676 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 38698a7b0cb249789d3f7f6e992b4015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:37 UTC) from {username='slave'} at 127.0.0.1:60290
I20260810 19:30:37.033228 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 18494
I20260810 19:30:37.046427 19242 tablet_replica.cc:330] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: stopping tablet replica
I20260810 19:30:37.046581 19243 tablet_replica.cc:330] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: stopping tablet replica
I20260810 19:30:37.046947 19242 raft_consensus.cc:2229] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:37.047077 19243 raft_consensus.cc:2229] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:37.047513 19242 raft_consensus.cc:2258] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:37.047571 19243 raft_consensus.cc:2258] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:37.048130 19241 tablet_replica.cc:330] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: stopping tablet replica
I20260810 19:30:37.048535 19241 raft_consensus.cc:2229] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:30:37.049206 19241 raft_consensus.cc:2258] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:37.050585 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.034663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.19.68.1:33791)
0810 19:30:37.034858 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:37.034866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:37.038504 (+ 3638us) server_negotiation.cc:407] Connection header received
0810 19:30:37.039368 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:37.039375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:37.039505 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:37.039585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:37.040790 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.042125 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.045760 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.046538 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.046663 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:37.048826 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:37.048837 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:37.048854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:37.049634 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:37.049694 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:37.049720 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:37.049806 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:37.050118 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 19:30:37.050235 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:30:37.050554 19243 ts_tablet_manager.cc:1905] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:37.051219 19242 ts_tablet_manager.cc:1905] T 38698a7b0cb249789d3f7f6e992b4015 P c37e486805f94d8dbc777fb726d17934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:37.051878 18709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:37.054277 19241 ts_tablet_manager.cc:1905] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:37.061039 19243 ts_tablet_manager.cc:1918] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:37.061290 19243 log.cc:1196] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/38698a7b0cb249789d3f7f6e992b4015
I20260810 19:30:37.061817 19243 ts_tablet_manager.cc:1939] T 38698a7b0cb249789d3f7f6e992b4015 P 5cbee539c7244c96aca1bcc7d09784b4: Deleting consensus metadata
I20260810 19:30:37.061803 19241 ts_tablet_manager.cc:1918] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:37.061964 19241 log.cc:1196] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/38698a7b0cb249789d3f7f6e992b4015
I20260810 19:30:37.062530 19241 ts_tablet_manager.cc:1939] T 38698a7b0cb249789d3f7f6e992b4015 P 135e99eb8ff14cf09eaa5fa3ad449430: Deleting consensus metadata
I20260810 19:30:37.063740 18371 catalog_manager.cc:4841] TS 5cbee539c7244c96aca1bcc7d09784b4 (127.19.68.3:43339): tablet 38698a7b0cb249789d3f7f6e992b4015 (table test-table [id=1240f0d8e8194be8b9b9dcaac5f0e5b6]) successfully deleted
I20260810 19:30:37.064191 18372 catalog_manager.cc:4841] TS 135e99eb8ff14cf09eaa5fa3ad449430 (127.19.68.2:39611): tablet 38698a7b0cb249789d3f7f6e992b4015 (table test-table [id=1240f0d8e8194be8b9b9dcaac5f0e5b6]) successfully deleted
I20260810 19:30:37.142521 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 18494
I20260810 19:30:37.172173 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 18725
W20260810 19:30:37.172165 18370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:33791 (error 108)
I20260810 19:30:37.174396 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.173827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60300)
0810 19:30:37.174089 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.174217 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260810 19:30:37.174651 18370 catalog_manager.cc:4584] TS c37e486805f94d8dbc777fb726d17934 (127.19.68.1:33791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38698a7b0cb249789d3f7f6e992b4015: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
I20260810 19:30:37.180423 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.173379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.19.68.2:39611)
0810 19:30:37.173566 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:37.173572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:37.173593 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:30:37.173659 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:37.173665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:37.173763 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:37.173887 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:37.174411 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.175578 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.176938 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.177663 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.177809 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:37.178804 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:37.178815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:37.178835 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:37.179630 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:37.179689 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:37.179713 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:37.179814 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:37.179940 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 19:30:37.180041 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:30:37.181192 18945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:37.228515 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.227964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60302)
0810 19:30:37.228202 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.228327 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 19:30:37.229722 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.229299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60314)
0810 19:30:37.229458 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.229547 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:30:37.290331 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.289773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60328)
0810 19:30:37.290005 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.290109 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 19:30:37.291388 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.291022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60336)
0810 19:30:37.291159 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.291232 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:30:37.295349 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 18725
I20260810 19:30:37.319226 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 18958
I20260810 19:30:37.325840 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.319978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.19.68.3:43339)
0810 19:30:37.320158 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:37.320163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:37.320195 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:30:37.320349 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:37.320353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:37.320455 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:37.320533 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:37.320883 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.321720 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.323030 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:37.323416 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:37.323510 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:37.324524 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:37.324533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:37.324547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:37.325151 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:37.325199 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:37.325227 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:37.325309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:37.325443 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 19:30:37.325525 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:30:37.326498 19155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:37.372500 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.371989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60338)
0810 19:30:37.372205 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.372344 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 19:30:37.373637 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:37.373225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:33791 (local address 127.0.0.1:60342)
0810 19:30:37.373425 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:37.373510 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:33791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 19:30:37.414384 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 18958
I20260810 19:30:37.437871 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 18342
I20260810 19:30:37.439060 18445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:37.560894 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 18342
2026-08-10T19:30:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2840 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:37 dist-test-slave-2j7r krb5kdc[19257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:37 dist-test-slave-2j7r krb5kdc[19257](info): set up 2 sockets
Aug 10 19:30:37 dist-test-slave-2j7r krb5kdc[19257](info): commencing operation
krb5kdc: starting...
W20260810 19:30:38.840229 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:39 dist-test-slave-2j7r krb5kdc[19257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:30:39.150607 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38509
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:39.490619 19272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:39.490937 19272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:39.491086 19272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:39.498317 19272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:39.498445 19272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:39.498530 19272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:39.498613 19272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:39.498674 19272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:39.508386 19272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:39.509871 19272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:39.511116 19272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:39.522353 19282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:39.522320 19283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:39.522289 19285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:39.523080 19272 server_base.cc:1034] running on GCE node
I20260810 19:30:39.523814 19272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:39.525118 19272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:39.526391 19272 hybrid_clock.cc:648] HybridClock initialized: now 1786390239526338 us; error 34 us; skew 500 ppm
I20260810 19:30:39.526726 19272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:39 dist-test-slave-2j7r krb5kdc[19257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:39.529717 19272 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:30:39.531162 19272 webserver.cc:460] Webserver started at http://127.19.68.62:43843/ using document root <none> and password file <none>
I20260810 19:30:39.531810 19272 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:39.531927 19272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:39.532203 19272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:39.535204 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "348a43fb8dcc4bb38e0232dbed739b21"
format_stamp: "Formatted at 2026-08-10 19:30:39 on dist-test-slave-2j7r"
server_key: "0d3a337988638601a94d8203fca717f1"
server_key_iv: "506b0cb25ba01949999085e1397eebdd"
server_key_version: "encryptionkey@0"
I20260810 19:30:39.535939 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "348a43fb8dcc4bb38e0232dbed739b21"
format_stamp: "Formatted at 2026-08-10 19:30:39 on dist-test-slave-2j7r"
server_key: "0d3a337988638601a94d8203fca717f1"
server_key_iv: "506b0cb25ba01949999085e1397eebdd"
server_key_version: "encryptionkey@0"
I20260810 19:30:39.540867 19272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:30:39.544148 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:39.545465 19272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:39.545621 19272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "348a43fb8dcc4bb38e0232dbed739b21"
format_stamp: "Formatted at 2026-08-10 19:30:39 on dist-test-slave-2j7r"
server_key: "0d3a337988638601a94d8203fca717f1"
server_key_iv: "506b0cb25ba01949999085e1397eebdd"
server_key_version: "encryptionkey@0"
I20260810 19:30:39.545791 19272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:39.563800 19272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:39.567529 19272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:39.567780 19272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:39.584896 19272 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:38509
I20260810 19:30:39.584919 19389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:38509 every 8 connection(s)
I20260810 19:30:39.586207 19272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:39.589758 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:39.590359 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19272
I20260810 19:30:39.590555 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:39.591159 19728 external_mini_cluster.cc:1428] Setting key 27101953a249ac2b8367a829d68d3ddb
I20260810 19:30:39.603343 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: Bootstrap starting.
Aug 10 19:30:39 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:39.607148 19391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:39.608281 19391 log.cc:824] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:39.611712 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: No bootstrap required, opened a new log
I20260810 19:30:39.616099 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:39.593000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.19.68.62:38509)
0810 19:30:39.593802 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:39.593813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:39.593855 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:30:39.595912 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:39.595933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:39.596346 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:39.596663 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:39.598509 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:39.599795 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:39.601856 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:39.602540 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:39.602673 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:39.607710 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:39.607718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:39.607732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:39.607771 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:39.611457 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:39.612239 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:39.612247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:39.612255 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:39.612482 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:39.612877 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:39.612882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:39.612885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:39.613638 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:39.613948 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:39.614341 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 19:30:39.614784 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":351,"threads_started":1}
I20260810 19:30:39.620812 19391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } }
I20260810 19:30:39.621181 19391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:39.621270 19391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348a43fb8dcc4bb38e0232dbed739b21, State: Initialized, Role: FOLLOWER
I20260810 19:30:39.621838 19391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } }
I20260810 19:30:39.622080 19391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:39.622212 19391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:39.622362 19391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:39.624781 19391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } }
I20260810 19:30:39.625253 19391 leader_election.cc:302] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348a43fb8dcc4bb38e0232dbed739b21; no voters:
I20260810 19:30:39.625849 19391 leader_election.cc:288] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:39.626013 19403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:39.626706 19403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [term 1 LEADER]: Becoming Leader. State: Replica: 348a43fb8dcc4bb38e0232dbed739b21, State: Running, Role: LEADER
I20260810 19:30:39.627249 19403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } }
I20260810 19:30:39.627779 19391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:39.630926 19406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348a43fb8dcc4bb38e0232dbed739b21. Latest consensus state: current_term: 1 leader_uuid: "348a43fb8dcc4bb38e0232dbed739b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } } }
I20260810 19:30:39.630980 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348a43fb8dcc4bb38e0232dbed739b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348a43fb8dcc4bb38e0232dbed739b21" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38509 } } }
I20260810 19:30:39.631407 19406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:39.631398 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:39.634670 19415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:39.640085 19415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:39.649045 19415 catalog_manager.cc:1347] Generated new cluster ID: 28ae0ea064784f3bb968d2cfacf8fb53
I20260810 19:30:39.649181 19415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:39.664218 19415 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:39.665480 19415 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:39.670747 19415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: Generated new TSK 0
I20260810 19:30:39.671473 19415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:30:39.766096 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:40.128684 19443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:40.128971 19443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:40.129109 19443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:40.136675 19443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:40.136785 19443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:40.136906 19443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:40.146610 19443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:40.148008 19443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:40.149397 19443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:40.161979 19450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:40.162014 19451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:40.162034 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:40.162890 19443 server_base.cc:1034] running on GCE node
I20260810 19:30:40.163465 19443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:40.164355 19443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:40.165560 19443 hybrid_clock.cc:648] HybridClock initialized: now 1786390240165495 us; error 46 us; skew 500 ppm
I20260810 19:30:40.165869 19443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:40 dist-test-slave-2j7r krb5kdc[19257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390240, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:40.168850 19443 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:30:40.170203 19443 webserver.cc:460] Webserver started at http://127.19.68.1:43613/ using document root <none> and password file <none>
I20260810 19:30:40.170930 19443 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:40.171044 19443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:40.171326 19443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:40.174230 19443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "3b8967cc9bd645f080cf6fecd4c44588"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "28487ebd0c38a61d0922a3868e8a6b4f"
server_key_iv: "9d27b2355cc92b416b692ded03eec532"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.174968 19443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "3b8967cc9bd645f080cf6fecd4c44588"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "28487ebd0c38a61d0922a3868e8a6b4f"
server_key_iv: "9d27b2355cc92b416b692ded03eec532"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.179823 19443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:30:40.183107 19466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.184331 19443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:30:40.184477 19443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "3b8967cc9bd645f080cf6fecd4c44588"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "28487ebd0c38a61d0922a3868e8a6b4f"
server_key_iv: "9d27b2355cc92b416b692ded03eec532"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.184646 19443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:40.197706 19443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:40.201344 19443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:40.201599 19443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:40.202489 19443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:40.203763 19443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:40.203850 19443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.203972 19443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:40.204030 19443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.234915 19443 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:40789
I20260810 19:30:40.234978 19676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:40789 every 8 connection(s)
I20260810 19:30:40.236255 19443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:40.241890 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19443
I20260810 19:30:40.242060 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:40.242525 19728 external_mini_cluster.cc:1428] Setting key 0262549726128c37230889aca4a04165
Aug 10 19:30:40 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390240, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:40.256134 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:40.238702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:43917 (local address 127.19.68.62:38509)
0810 19:30:40.239157 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:40.239163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:40.240115 (+ 952us) server_negotiation.cc:407] Connection header received
0810 19:30:40.241948 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:40.241954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:40.242048 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:40.242152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:40.243874 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.244822 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.246408 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.246958 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.247075 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:40.250666 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:40.250671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:40.250674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:40.250707 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:40.253250 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:40.254041 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:40.254044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:40.254047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:40.254159 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:40.254581 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:40.254584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:40.254586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:40.254825 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:40.254935 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:40.255704 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 19:30:40.255873 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
I20260810 19:30:40.257429 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:40.239236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38509 (local address 127.19.68.1:43917)
0810 19:30:40.239953 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:40.239997 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:40.241685 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:40.242414 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:40.242422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:40.242970 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:40.243693 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:40.243718 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.244993 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.244997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:40.246262 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:40.246275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.247147 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.247151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:40.247338 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:40.247951 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:40.247974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:40.250440 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:40.253394 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:40.253402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:40.253417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:40.253941 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:40.254247 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:40.254252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:40.254255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:40.254492 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:40.254938 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:40.254945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:40.255344 (+ 399us) client_negotiation.cc:769] Sending connection context
0810 19:30:40.255613 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 19:30:40.256113 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260810 19:30:40.258883 19680 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38509
I20260810 19:30:40.259207 19680 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:40.260082 19680 heartbeater.cc:511] Master 127.19.68.62:38509 requested a full tablet report, sending...
I20260810 19:30:40.262166 19330 ts_manager.cc:194] Registered new tserver with Master: 3b8967cc9bd645f080cf6fecd4c44588 (127.19.68.1:40789)
I20260810 19:30:40.263549 19330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:43917
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:30:40.310088 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:40.653712 19689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:40.653991 19689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:40.654139 19689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:40.661504 19689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:40.661615 19689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:40.661787 19689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:40.671630 19689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:40.672873 19689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:40.674230 19689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:40.687445 19701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:40.687522 19697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:40.687522 19698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:40.688257 19689 server_base.cc:1034] running on GCE node
I20260810 19:30:40.688831 19689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:40.689801 19689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:40.691007 19689 hybrid_clock.cc:648] HybridClock initialized: now 1786390240690958 us; error 45 us; skew 500 ppm
I20260810 19:30:40.691306 19689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:40 dist-test-slave-2j7r krb5kdc[19257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390240, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:40.694403 19689 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:30:40.695789 19689 webserver.cc:460] Webserver started at http://127.19.68.2:42869/ using document root <none> and password file <none>
I20260810 19:30:40.696444 19689 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:40.696564 19689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:40.696856 19689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:40.699661 19689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "a812568279ac43ca8af8a85d9fbb6658"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "ee03d6214237f151f81232cce174e97c"
server_key_iv: "f80667612dd9a7c375fdfd167fe1b0ad"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.700397 19689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "a812568279ac43ca8af8a85d9fbb6658"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "ee03d6214237f151f81232cce174e97c"
server_key_iv: "f80667612dd9a7c375fdfd167fe1b0ad"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.705309 19689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:30:40.708685 19711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.710007 19689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:40.710155 19689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "a812568279ac43ca8af8a85d9fbb6658"
format_stamp: "Formatted at 2026-08-10 19:30:40 on dist-test-slave-2j7r"
server_key: "ee03d6214237f151f81232cce174e97c"
server_key_iv: "f80667612dd9a7c375fdfd167fe1b0ad"
server_key_version: "encryptionkey@0"
I20260810 19:30:40.710356 19689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:40.729173 19689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:40.732913 19689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:40.733214 19689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:40.734244 19689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:40.735551 19689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:40.735630 19689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.735750 19689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:40.735805 19689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:40.765866 19689 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:44751
I20260810 19:30:40.765928 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:44751 every 8 connection(s)
I20260810 19:30:40.767109 19689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:40.770082 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19689
I20260810 19:30:40.770256 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:40.770845 19728 external_mini_cluster.cc:1428] Setting key c429fc0b681ddb7bd23818e6cb5ec356
Aug 10 19:30:40 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390240, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:40.788467 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:40.769494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:41047 (local address 127.19.68.62:38509)
0810 19:30:40.770025 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:40.770031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:40.771070 (+ 1039us) server_negotiation.cc:407] Connection header received
0810 19:30:40.773208 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:40.773212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:40.773278 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:40.773390 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:40.775314 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.776060 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.778185 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.778618 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.778724 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:40.782692 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:40.782697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:40.782700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:40.782732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:40.785230 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:40.786024 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:40.786027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:40.786030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:40.786157 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:40.786570 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:40.786573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:40.786575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:40.786818 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:40.786933 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:40.787795 (+ 862us) server_negotiation.cc:299] Negotiation successful
0810 19:30:40.788036 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
I20260810 19:30:40.789414 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:40.770031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38509 (local address 127.19.68.2:41047)
0810 19:30:40.770869 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:40.770923 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:40.772901 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:40.773645 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:40.773657 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:40.774361 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:40.775136 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:40.775160 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.776222 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.776226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:40.778007 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:40.778023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:40.778855 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:40.778862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:40.779128 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:40.779798 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:40.779826 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:40.782463 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:40.785367 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:40.785375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:40.785391 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:40.785906 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:40.786248 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:40.786252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:40.786255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:40.786480 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:40.786933 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:40.786940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:40.787356 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 19:30:40.787659 (+ 303us) client_negotiation.cc:240] Negotiation successful
0810 19:30:40.788088 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":331,"threads_started":1}
I20260810 19:30:40.790750 19929 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38509
I20260810 19:30:40.791142 19929 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:40.792156 19929 heartbeater.cc:511] Master 127.19.68.62:38509 requested a full tablet report, sending...
I20260810 19:30:40.793922 19330 ts_manager.cc:194] Registered new tserver with Master: a812568279ac43ca8af8a85d9fbb6658 (127.19.68.2:44751)
I20260810 19:30:40.794740 19330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:41047
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:30:40.840008 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:41.179829 19941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:41.180096 19941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:41.180197 19941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:41.187312 19941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:41.187440 19941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:41.187574 19941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:41.197104 19941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:41.198315 19941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:41.199582 19941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:41.212503 19953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:41.212540 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:41.212564 19952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:41.213619 19941 server_base.cc:1034] running on GCE node
I20260810 19:30:41.214200 19941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:41.215193 19941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:41.216434 19941 hybrid_clock.cc:648] HybridClock initialized: now 1786390241216378 us; error 36 us; skew 500 ppm
I20260810 19:30:41.216755 19941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:41 dist-test-slave-2j7r krb5kdc[19257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390241, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:41.219852 19941 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:30:41.221262 19941 webserver.cc:460] Webserver started at http://127.19.68.3:42493/ using document root <none> and password file <none>
I20260810 19:30:41.221913 19941 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:41.222019 19941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:41.222283 19941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:41.225194 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "fd288cef16f9434693eb94c8b35a3a76"
format_stamp: "Formatted at 2026-08-10 19:30:41 on dist-test-slave-2j7r"
server_key: "966e6c1a708e9ad373de22e8d9684726"
server_key_iv: "8e31ccb4fc1797b66f0a7ea64c1aa2d2"
server_key_version: "encryptionkey@0"
I20260810 19:30:41.225957 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "fd288cef16f9434693eb94c8b35a3a76"
format_stamp: "Formatted at 2026-08-10 19:30:41 on dist-test-slave-2j7r"
server_key: "966e6c1a708e9ad373de22e8d9684726"
server_key_iv: "8e31ccb4fc1797b66f0a7ea64c1aa2d2"
server_key_version: "encryptionkey@0"
I20260810 19:30:41.231279 19941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:30:41.235028 19979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:41.236426 19941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:30:41.236582 19941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "fd288cef16f9434693eb94c8b35a3a76"
format_stamp: "Formatted at 2026-08-10 19:30:41 on dist-test-slave-2j7r"
server_key: "966e6c1a708e9ad373de22e8d9684726"
server_key_iv: "8e31ccb4fc1797b66f0a7ea64c1aa2d2"
server_key_version: "encryptionkey@0"
I20260810 19:30:41.236761 19941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:41.266875 19680 heartbeater.cc:503] Master 127.19.68.62:38509 was elected leader, sending a full tablet report...
I20260810 19:30:41.278216 19941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:41.282179 19941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:41.282464 19941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:41.283419 19941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:41.284741 19941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:41.284828 19941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:41.284953 19941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:41.285014 19941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:41.315428 19941 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:38091
I20260810 19:30:41.315503 20192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:38091 every 8 connection(s)
I20260810 19:30:41.316632 19941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:41.326347 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 19941
I20260810 19:30:41.326560 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:41.327071 19728 external_mini_cluster.cc:1428] Setting key bc4446305aa4b0f959f408c2f3426d0c
Aug 10 19:30:41 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390241, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:41.336059 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.319047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:56483 (local address 127.19.68.62:38509)
0810 19:30:41.319508 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.319512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.320446 (+ 934us) server_negotiation.cc:407] Connection header received
0810 19:30:41.322276 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.322283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.322395 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.322497 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:41.323976 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.324763 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.326464 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.326887 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.326990 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.330893 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:41.330898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:41.330901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:41.330933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:41.333524 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.334169 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.334173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.334175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.334293 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.334671 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.334674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.334676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.334959 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:41.335070 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.335641 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.335801 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":210,"threads_started":1}
I20260810 19:30:41.336935 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.319582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38509 (local address 127.19.68.3:56483)
0810 19:30:41.320282 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.320327 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.322014 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.322752 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.322760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.323269 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:41.323811 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.323827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.324914 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.324918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.326304 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.326326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.327110 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.327118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.327389 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.328067 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:41.328098 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:41.330700 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:41.333652 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:41.333658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:41.333670 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:41.334081 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:41.334376 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:41.334379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:41.334381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:41.334591 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:41.335059 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:41.335065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:41.335349 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.335557 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.335910 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260810 19:30:41.338241 20194 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38509
I20260810 19:30:41.338488 20194 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:41.339134 20194 heartbeater.cc:511] Master 127.19.68.62:38509 requested a full tablet report, sending...
I20260810 19:30:41.340495 19330 ts_manager.cc:194] Registered new tserver with Master: fd288cef16f9434693eb94c8b35a3a76 (127.19.68.3:38091)
I20260810 19:30:41.341544 19330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:56483
I20260810 19:30:41.342254 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:41.358223 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.346199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.19.68.62:38509)
0810 19:30:41.346379 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.346384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.346783 (+ 399us) server_negotiation.cc:407] Connection header received
0810 19:30:41.346949 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.346954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.347020 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.347096 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:41.348530 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.349284 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.350617 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.350992 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.351089 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.353518 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:41.353523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:41.353526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:41.353556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:41.355957 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.356603 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.356606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.356608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.356718 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.357048 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.357050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.357052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.357404 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:41.357600 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.357876 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.358004 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:30:41.361956 19328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:41.368677 19328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:41.397684 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.385688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.19.68.1:40789)
0810 19:30:41.389123 (+ 3435us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.389133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.389569 (+ 436us) server_negotiation.cc:407] Connection header received
0810 19:30:41.389677 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.389685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.389946 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.390910 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:41.391151 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.392677 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.395535 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.396537 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.396662 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.396929 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.397187 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.397358 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3232,"thread_start_us":3078,"threads_started":1}
I20260810 19:30:41.397738 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.380526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49886)
0810 19:30:41.389069 (+ 8543us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.389145 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.389367 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.390193 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.390200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.390236 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:41.390712 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.390728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.392840 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.392845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.395346 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.395368 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.396740 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.396746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.396929 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.396982 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.397134 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.397339 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8241,"thread_start_us":8566,"threads_started":1}
I20260810 19:30:41.402086 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.390168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44751 (local address 127.0.0.1:60000)
0810 19:30:41.394033 (+ 3865us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.394054 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.394178 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.395068 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.395072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.395091 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:41.395455 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.395466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.397346 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.397352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.399901 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.399917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.401204 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.401210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.401397 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.401429 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.401526 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.401656 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3750,"mutex_wait_us":371,"thread_start_us":3598,"threads_started":1}
I20260810 19:30:41.402343 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.385696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.19.68.3:38091)
0810 19:30:41.393623 (+ 7927us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.393630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.393664 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:30:41.393756 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.393764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.394006 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.395047 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:41.395341 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.397413 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.399972 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.401323 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.401457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.401675 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.401862 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.402026 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7745,"thread_start_us":7596,"threads_started":1}
I20260810 19:30:41.402730 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.380785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38091 (local address 127.0.0.1:55122)
0810 19:30:41.389407 (+ 8622us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.389423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.389533 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.394234 (+ 4701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.394239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.394271 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:41.394692 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.394704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.397793 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.397799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.399770 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.399783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.401523 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.401529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.401693 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.401722 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.401817 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.401949 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8510,"mutex_wait_us":4,"thread_start_us":8330,"threads_started":1}
I20260810 19:30:41.402846 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.390251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.19.68.2:44751)
0810 19:30:41.391309 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.391318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.394305 (+ 2987us) server_negotiation.cc:407] Connection header received
0810 19:30:41.394429 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.394436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.394658 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.394851 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:41.395639 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.397158 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.400057 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.400987 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.401123 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.401317 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.402353 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.402534 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":2504,"threads_started":1}
I20260810 19:30:41.403026 19558 tablet_service.cc:1467] Processing CreateTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 (DEFAULT_TABLE table=test-table [id=6bc8a09bb1ac41fb97cbee8eccb11d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:41.404670 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:41.406327 20072 tablet_service.cc:1467] Processing CreateTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 (DEFAULT_TABLE table=test-table [id=6bc8a09bb1ac41fb97cbee8eccb11d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:41.406654 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 (DEFAULT_TABLE table=test-table [id=6bc8a09bb1ac41fb97cbee8eccb11d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:41.407764 20072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:41.407992 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:41.419116 20223 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Bootstrap starting.
I20260810 19:30:41.422803 20223 tablet_bootstrap.cc:654] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:41.423940 20225 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Bootstrap starting.
I20260810 19:30:41.424096 20223 log.cc:824] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:41.425581 20224 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Bootstrap starting.
I20260810 19:30:41.427911 20225 tablet_bootstrap.cc:654] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:41.429155 20225 log.cc:824] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:41.429320 20224 tablet_bootstrap.cc:654] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:41.429443 20223 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: No bootstrap required, opened a new log
I20260810 19:30:41.429728 20223 ts_tablet_manager.cc:1397] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260810 19:30:41.430552 20224 log.cc:824] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:41.431998 20225 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: No bootstrap required, opened a new log
I20260810 19:30:41.432240 20225 ts_tablet_manager.cc:1397] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 19:30:41.433597 20224 tablet_bootstrap.cc:492] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: No bootstrap required, opened a new log
I20260810 19:30:41.433861 20224 ts_tablet_manager.cc:1397] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 19:30:41.440107 20225 raft_consensus.cc:348] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.440124 20223 raft_consensus.cc:348] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.440529 20225 raft_consensus.cc:374] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:41.440591 20223 raft_consensus.cc:374] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:41.440620 20225 raft_consensus.cc:729] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a812568279ac43ca8af8a85d9fbb6658, State: Initialized, Role: FOLLOWER
I20260810 19:30:41.440689 20223 raft_consensus.cc:729] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd288cef16f9434693eb94c8b35a3a76, State: Initialized, Role: FOLLOWER
I20260810 19:30:41.441354 20225 consensus_queue.cc:260] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.441476 20223 consensus_queue.cc:260] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.442530 19929 heartbeater.cc:503] Master 127.19.68.62:38509 was elected leader, sending a full tablet report...
I20260810 19:30:41.443279 20225 ts_tablet_manager.cc:1428] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 19:30:41.444219 20194 heartbeater.cc:503] Master 127.19.68.62:38509 was elected leader, sending a full tablet report...
I20260810 19:30:41.444693 20224 raft_consensus.cc:348] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.445223 20224 raft_consensus.cc:374] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:41.445331 20224 raft_consensus.cc:729] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8967cc9bd645f080cf6fecd4c44588, State: Initialized, Role: FOLLOWER
I20260810 19:30:41.446087 20224 consensus_queue.cc:260] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.446627 20223 ts_tablet_manager.cc:1428] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260810 19:30:41.448553 20224 ts_tablet_manager.cc:1428] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260810 19:30:41.473318 20236 raft_consensus.cc:482] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:41.473663 20236 raft_consensus.cc:504] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.475442 20236 leader_election.cc:288] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd288cef16f9434693eb94c8b35a3a76 (127.19.68.3:38091), a812568279ac43ca8af8a85d9fbb6658 (127.19.68.2:44751)
I20260810 19:30:41.482486 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.475892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:56291 (local address 127.19.68.3:38091)
0810 19:30:41.476122 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.476130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.476841 (+ 711us) server_negotiation.cc:407] Connection header received
0810 19:30:41.477022 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.477029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.477124 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.477205 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:41.477881 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.478721 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.480707 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.481541 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.481655 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.481761 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.482119 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.482244 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:30:41.482520 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.476246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:53873 (local address 127.19.68.2:44751)
0810 19:30:41.476439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.476445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.476749 (+ 304us) server_negotiation.cc:407] Connection header received
0810 19:30:41.476964 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.476968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.477031 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.477123 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:41.477776 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.478549 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.480711 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.481540 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.481669 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.481779 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.482133 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.482262 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:30:41.482806 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.475734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:44751 (local address 127.19.68.1:53873)
0810 19:30:41.476635 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.476658 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.476796 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.477162 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.477166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.477191 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:41.477604 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.477616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.478706 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.478711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.480554 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.480566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.481817 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.481823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.481973 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.482005 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.482088 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.482212 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":9,"spinlock_wait_cycles":23296,"thread_start_us":552,"threads_started":1}
I20260810 19:30:41.482956 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.475785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:38091 (local address 127.19.68.1:56291)
0810 19:30:41.476744 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.476762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:41.476894 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:41.477233 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:41.477237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:41.477265 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:41.477700 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.477710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.478881 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.478886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.480555 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:41.480568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.481773 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.481780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:41.481936 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.481993 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 19:30:41.482074 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:30:41.482196 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":52,"thread_start_us":684,"threads_started":1}
I20260810 19:30:41.483718 19839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c99d0f4ac4c91afc6bd2ea2e14a67" candidate_uuid: "3b8967cc9bd645f080cf6fecd4c44588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812568279ac43ca8af8a85d9fbb6658" is_pre_election: true
I20260810 19:30:41.483837 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c99d0f4ac4c91afc6bd2ea2e14a67" candidate_uuid: "3b8967cc9bd645f080cf6fecd4c44588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd288cef16f9434693eb94c8b35a3a76" is_pre_election: true
I20260810 19:30:41.484200 19839 raft_consensus.cc:2454] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8967cc9bd645f080cf6fecd4c44588 in term 0.
I20260810 19:30:41.484308 20106 raft_consensus.cc:2454] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8967cc9bd645f080cf6fecd4c44588 in term 0.
I20260810 19:30:41.484834 19472 leader_election.cc:302] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8967cc9bd645f080cf6fecd4c44588, a812568279ac43ca8af8a85d9fbb6658; no voters:
I20260810 19:30:41.485209 20236 raft_consensus.cc:2781] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:41.485363 20236 raft_consensus.cc:482] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:41.485471 20236 raft_consensus.cc:3037] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:41.487644 20236 raft_consensus.cc:504] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.488198 20236 leader_election.cc:288] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [CANDIDATE]: Term 1 election: Requested vote from peers fd288cef16f9434693eb94c8b35a3a76 (127.19.68.3:38091), a812568279ac43ca8af8a85d9fbb6658 (127.19.68.2:44751)
W20260810 19:30:41.488504 19681 tablet.cc:2369] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:41.488682 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c99d0f4ac4c91afc6bd2ea2e14a67" candidate_uuid: "3b8967cc9bd645f080cf6fecd4c44588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd288cef16f9434693eb94c8b35a3a76"
I20260810 19:30:41.488837 20106 raft_consensus.cc:3037] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:41.489542 19839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c99d0f4ac4c91afc6bd2ea2e14a67" candidate_uuid: "3b8967cc9bd645f080cf6fecd4c44588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812568279ac43ca8af8a85d9fbb6658"
I20260810 19:30:41.489751 19839 raft_consensus.cc:3037] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:41.491084 20106 raft_consensus.cc:2454] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8967cc9bd645f080cf6fecd4c44588 in term 1.
I20260810 19:30:41.491437 19473 leader_election.cc:302] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8967cc9bd645f080cf6fecd4c44588, fd288cef16f9434693eb94c8b35a3a76; no voters:
I20260810 19:30:41.491698 20236 raft_consensus.cc:2781] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:41.492249 20236 raft_consensus.cc:686] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8967cc9bd645f080cf6fecd4c44588, State: Running, Role: LEADER
I20260810 19:30:41.492323 19839 raft_consensus.cc:2454] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8967cc9bd645f080cf6fecd4c44588 in term 1.
I20260810 19:30:41.492830 20236 consensus_queue.cc:237] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } }
I20260810 19:30:41.497665 19330 catalog_manager.cc:5495] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8967cc9bd645f080cf6fecd4c44588 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "3b8967cc9bd645f080cf6fecd4c44588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8967cc9bd645f080cf6fecd4c44588" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 } health_report { overall_health: UNKNOWN } } }
W20260810 19:30:41.519038 19934 tablet.cc:2369] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:41.534477 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.529616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.19.68.1:40789)
0810 19:30:41.529784 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.529789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.529807 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:30:41.529898 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.529902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.529994 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.530120 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:41.530424 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.531142 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.532391 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.532787 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.532886 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.533438 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:41.533954 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:41.534087 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.534231 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.534277 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:30:41.542420 19839 raft_consensus.cc:1261] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 1 FOLLOWER]: Refusing update from remote peer 3b8967cc9bd645f080cf6fecd4c44588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:41.542420 20106 raft_consensus.cc:1261] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 1 FOLLOWER]: Refusing update from remote peer 3b8967cc9bd645f080cf6fecd4c44588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:41.543407 20242 consensus_queue.cc:1035] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd288cef16f9434693eb94c8b35a3a76" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:41.543723 20236 consensus_queue.cc:1035] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a812568279ac43ca8af8a85d9fbb6658" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:41.553983 20247 mvcc.cc:205] Tried to move back new op lower bound from 7317054429347405824 to 7317054429158445056. Current Snapshot: MvccSnapshot[applied={T|T < 7317054429347405824}]
I20260810 19:30:41.583845 19328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:41.584165 19328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:41.588224 19328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 348a43fb8dcc4bb38e0232dbed739b21: Sending DeleteTablet for 3 replicas of tablet 899c99d0f4ac4c91afc6bd2ea2e14a67
I20260810 19:30:41.589366 19558 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:41 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49886
I20260810 19:30:41.590055 20072 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:41 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:55122
I20260810 19:30:41.590165 20268 tablet_replica.cc:330] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: stopping tablet replica
I20260810 19:30:41.590200 19805 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:41 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 19:30:41.590756 20268 raft_consensus.cc:2229] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:30:41.591182 19556 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:41 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:49886
I20260810 19:30:41.591560 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19443
I20260810 19:30:41.591527 20268 raft_consensus.cc:2258] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:41.600198 20271 tablet_replica.cc:330] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: stopping tablet replica
I20260810 19:30:41.600600 20271 raft_consensus.cc:2229] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:41.601184 20271 raft_consensus.cc:2258] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:41.601949 20272 tablet_replica.cc:330] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: stopping tablet replica
I20260810 19:30:41.602329 20272 raft_consensus.cc:2229] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:41.602593 20268 ts_tablet_manager.cc:1905] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P 3b8967cc9bd645f080cf6fecd4c44588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:41.602821 20272 raft_consensus.cc:2258] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:41.604792 20272 ts_tablet_manager.cc:1905] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:41.604876 20271 ts_tablet_manager.cc:1905] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:30:41 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:30:41.611460 20272 ts_tablet_manager.cc:1918] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:41.611603 20272 log.cc:1196] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/899c99d0f4ac4c91afc6bd2ea2e14a67
I20260810 19:30:41.612095 20272 ts_tablet_manager.cc:1939] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P a812568279ac43ca8af8a85d9fbb6658: Deleting consensus metadata
I20260810 19:30:41.613502 20271 ts_tablet_manager.cc:1918] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:41.613682 20271 log.cc:1196] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/899c99d0f4ac4c91afc6bd2ea2e14a67
I20260810 19:30:41.614040 19304 catalog_manager.cc:4841] TS a812568279ac43ca8af8a85d9fbb6658 (127.19.68.2:44751): tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 (table test-table [id=6bc8a09bb1ac41fb97cbee8eccb11d0d]) successfully deleted
I20260810 19:30:41.614207 20271 ts_tablet_manager.cc:1939] T 899c99d0f4ac4c91afc6bd2ea2e14a67 P fd288cef16f9434693eb94c8b35a3a76: Deleting consensus metadata
I20260810 19:30:41.615862 19305 catalog_manager.cc:4841] TS fd288cef16f9434693eb94c8b35a3a76 (127.19.68.3:38091): tablet 899c99d0f4ac4c91afc6bd2ea2e14a67 (table test-table [id=6bc8a09bb1ac41fb97cbee8eccb11d0d]) successfully deleted
I20260810 19:30:41.616914 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.594885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.19.68.1:40789)
0810 19:30:41.595035 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.595041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.595060 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:30:41.595108 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.595113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.595180 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.595273 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:41.597164 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.598375 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.600395 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.600986 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.601110 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.608026 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:41.608041 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:41.608057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:41.608110 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:41.612982 (+ 4872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.614288 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.614306 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.614310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.614475 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.615002 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.615007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.615010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.615448 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:41.615691 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.616296 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.616540 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:30:41.618312 19666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:41.711520 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19443
I20260810 19:30:41.735154 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19689
W20260810 19:30:41.735122 19304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:40789 (error 108)
I20260810 19:30:41.736958 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.736528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49924)
0810 19:30:41.736664 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.736782 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 19:30:41.737227 19304 catalog_manager.cc:4584] TS 3b8967cc9bd645f080cf6fecd4c44588 (127.19.68.1:40789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 899c99d0f4ac4c91afc6bd2ea2e14a67: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Aug 10 19:30:41 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:30:41.749815 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.735964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.19.68.2:44751)
0810 19:30:41.736245 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.736250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.736293 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:30:41.736535 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.736540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.736630 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.736755 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:41.738177 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.738991 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.740278 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.740654 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.740747 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.744718 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:41.744732 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:41.744746 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:41.744779 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:41.747311 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.747986 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.747992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.747995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.748107 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.748409 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.748412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.748414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.748800 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:41.749035 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.749330 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.749501 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 19:30:41.750454 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:41.781889 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.781420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49934)
0810 19:30:41.781628 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.781725 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 19:30:41.782963 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.782598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49942)
0810 19:30:41.782753 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.782828 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 19:30:41.790982 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.790619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49954)
0810 19:30:41.790782 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.790850 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:30:41.791855 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.791555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49966)
0810 19:30:41.791685 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.791748 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 19:30:41.832193 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.831690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49980)
0810 19:30:41.831895 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.832001 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 19:30:41.833258 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.832800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:49994)
0810 19:30:41.833021 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.833139 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260810 19:30:41.837630 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19689
I20260810 19:30:41.863142 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19941
I20260810 19:30:41.869119 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.868572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50006)
0810 19:30:41.868777 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.868877 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 19:30:41.870113 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.869750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50014)
0810 19:30:41.869922 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.870002 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Aug 10 19:30:41 dist-test-slave-2j7r krb5kdc[19257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:30:41.877924 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.863922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.19.68.3:38091)
0810 19:30:41.864149 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:41.864155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:41.864175 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:30:41.864271 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:41.864277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:41.864368 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:41.864455 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:41.865860 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.866606 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.867896 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:41.868398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:41.868514 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:41.872889 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:41.872904 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:41.872920 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:41.872956 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:41.875634 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.876213 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.876219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.876221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.876367 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:41.876645 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:41.876648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:41.876650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:41.876998 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:41.877246 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:41.877489 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:30:41.877658 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:30:41.878551 20186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:41.934608 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.934077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50030)
0810 19:30:41.934316 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.934426 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:30:41.935701 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.935347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50040)
0810 19:30:41.935488 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.935571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:30:41.966687 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19941
I20260810 19:30:41.972484 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.972050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50048)
0810 19:30:41.972214 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.972292 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:30:41.973542 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:41.973219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40789 (local address 127.0.0.1:50050)
0810 19:30:41.973326 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:41.973399 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 19:30:41.994325 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 19272
I20260810 19:30:41.995415 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:42.105007 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 19272
2026-08-10T19:30:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4541 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T19:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:42Z Disabled control of system clock
I20260810 19:30:42.199759 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35091
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:42.535892 20286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:42.536187 20286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:42.536324 20286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:42.543789 20286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:42.543921 20286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:42.543985 20286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:42.544065 20286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:42.544117 20286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:42.553622 20286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:42.554981 20286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:42.556946 20286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:42.568470 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:42.568399 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:42.568472 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:42.569144 20286 server_base.cc:1034] running on GCE node
I20260810 19:30:42.569877 20286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:42.571192 20286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:42.572423 20286 hybrid_clock.cc:648] HybridClock initialized: now 1786390242572366 us; error 41 us; skew 500 ppm
I20260810 19:30:42.572767 20286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:42.575040 20286 webserver.cc:460] Webserver started at http://127.19.68.62:39299/ using document root <none> and password file <none>
I20260810 19:30:42.575687 20286 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:42.575793 20286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:42.576105 20286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:42.578984 20286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "7ffd87ef1c15461b915fbfc4c12e1da0"
format_stamp: "Formatted at 2026-08-10 19:30:42 on dist-test-slave-2j7r"
server_key: "8609b8114192118ef8e1618de4b638bd"
server_key_iv: "2c2d88f0717749cc1c7ad6b25648e165"
server_key_version: "encryptionkey@0"
I20260810 19:30:42.579747 20286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "7ffd87ef1c15461b915fbfc4c12e1da0"
format_stamp: "Formatted at 2026-08-10 19:30:42 on dist-test-slave-2j7r"
server_key: "8609b8114192118ef8e1618de4b638bd"
server_key_iv: "2c2d88f0717749cc1c7ad6b25648e165"
server_key_version: "encryptionkey@0"
I20260810 19:30:42.584679 20286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:30:42.588105 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:42.589422 20286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:42.589576 20286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "7ffd87ef1c15461b915fbfc4c12e1da0"
format_stamp: "Formatted at 2026-08-10 19:30:42 on dist-test-slave-2j7r"
server_key: "8609b8114192118ef8e1618de4b638bd"
server_key_iv: "2c2d88f0717749cc1c7ad6b25648e165"
server_key_version: "encryptionkey@0"
I20260810 19:30:42.589767 20286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:42.601119 20286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:42.601871 20286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:42.602128 20286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:42.619285 20286 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:35091
I20260810 19:30:42.619292 20408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:35091 every 8 connection(s)
I20260810 19:30:42.620520 20286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:42.624073 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:42.628871 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20286
I20260810 19:30:42.629066 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:42.629698 19728 external_mini_cluster.cc:1428] Setting key ac23923b6bb83ba4d2cb4ba7ce9c1297
I20260810 19:30:42.635159 20409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: Bootstrap starting.
I20260810 19:30:42.638600 20409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:42.639704 20409 log.cc:824] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:42.643028 20409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: No bootstrap required, opened a new log
I20260810 19:30:42.643249 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:42.631598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.19.68.62:35091)
0810 19:30:42.632587 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:42.632597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:42.632639 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:30:42.634355 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:42.634384 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:42.634809 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:42.635116 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:42.635704 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:42.637018 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:42.638961 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:42.639612 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:42.639745 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:42.640982 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:42.640992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:42.641009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:42.641653 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:42.641725 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:42.641746 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:42.641842 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:42.642038 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 19:30:42.642309 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260810 19:30:42.650581 20409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } }
I20260810 19:30:42.650899 20409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:42.651005 20409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd87ef1c15461b915fbfc4c12e1da0, State: Initialized, Role: FOLLOWER
I20260810 19:30:42.651551 20409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } }
I20260810 19:30:42.651774 20409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:42.651907 20409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:42.652132 20409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:42.654284 20409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } }
I20260810 19:30:42.654719 20409 leader_election.cc:302] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ffd87ef1c15461b915fbfc4c12e1da0; no voters:
I20260810 19:30:42.655239 20409 leader_election.cc:288] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:42.655356 20416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:42.656000 20416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffd87ef1c15461b915fbfc4c12e1da0, State: Running, Role: LEADER
I20260810 19:30:42.656508 20416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } }
I20260810 19:30:42.657146 20409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:42.659519 20420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ffd87ef1c15461b915fbfc4c12e1da0. Latest consensus state: current_term: 1 leader_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } } }
I20260810 19:30:42.659899 20420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:42.661525 20431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:42.661371 20418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffd87ef1c15461b915fbfc4c12e1da0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35091 } } }
I20260810 19:30:42.661667 20418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:42.666870 20431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:42.676267 20431 catalog_manager.cc:1347] Generated new cluster ID: 912317a6c160440dacabaf5e32bc28c9
I20260810 19:30:42.676369 20431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:42.688478 20431 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:42.689736 20431 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:42.700284 20431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: Generated new TSK 0
I20260810 19:30:42.701026 20431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:42.708484 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--builtin_ntp_servers=127.19.68.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:43.047775 20456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:43.048041 20456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:43.048135 20456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:43.055164 20456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:43.055263 20456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:43.055382 20456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:43.064483 20456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:43.065708 20456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:43.067474 20456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:43.079946 20463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.079990 20464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.079906 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:43.080808 20456 server_base.cc:1034] running on GCE node
I20260810 19:30:43.081435 20456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:43.082355 20456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:43.083554 20456 hybrid_clock.cc:648] HybridClock initialized: now 1786390243083500 us; error 48 us; skew 500 ppm
I20260810 19:30:43.083850 20456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:43.090844 20456 webserver.cc:460] Webserver started at http://127.19.68.1:38593/ using document root <none> and password file <none>
I20260810 19:30:43.091497 20456 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:43.091601 20456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:43.091902 20456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:43.094928 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "058763074889426c9748d7d110233eaf"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "d4b32ba61681cde918cd6e0fd7deaa65"
server_key_iv: "9fa3c06618b5fd39799861d4b9169645"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.095685 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "058763074889426c9748d7d110233eaf"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "d4b32ba61681cde918cd6e0fd7deaa65"
server_key_iv: "9fa3c06618b5fd39799861d4b9169645"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.100616 20456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:30:43.103960 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.105325 20456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:30:43.105476 20456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "058763074889426c9748d7d110233eaf"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "d4b32ba61681cde918cd6e0fd7deaa65"
server_key_iv: "9fa3c06618b5fd39799861d4b9169645"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.105664 20456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:43.122952 20456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:43.123783 20456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:43.124064 20456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:43.125108 20456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:43.126586 20456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:43.126688 20456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.126797 20456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:43.126891 20456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.157196 20456 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:38367
I20260810 19:30:43.157249 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:38367 every 8 connection(s)
I20260810 19:30:43.158557 20456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:43.165444 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20456
I20260810 19:30:43.165613 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:43.166064 19728 external_mini_cluster.cc:1428] Setting key fe99018c3cabe7c332e74425fdf4804f
I20260810 19:30:43.168618 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--builtin_ntp_servers=127.19.68.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:43.170593 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:43.160890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:33053 (local address 127.19.68.62:35091)
0810 19:30:43.161329 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:43.161333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:43.162506 (+ 1173us) server_negotiation.cc:407] Connection header received
0810 19:30:43.164310 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:43.164316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:43.164395 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:43.164518 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:43.165656 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.166441 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.167968 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.168355 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.168464 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:43.169790 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:43.169795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:43.169798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:43.169816 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:43.169860 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:43.169872 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:43.169955 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:43.170171 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 19:30:43.170268 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260810 19:30:43.171546 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:43.161466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35091 (local address 127.19.68.1:33053)
0810 19:30:43.162339 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:43.162374 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:43.164054 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:43.164766 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:43.164774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:43.164857 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:43.165485 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:43.165505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.166599 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.166603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:43.167822 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:43.167834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.168576 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.168583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:43.168827 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:43.169500 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:43.169503 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:43.169541 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:43.169618 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:43.169961 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:43.169973 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:43.169981 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:30:43.170126 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 19:30:43.170522 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":383,"threads_started":1}
I20260810 19:30:43.173004 20671 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35091
I20260810 19:30:43.173305 20671 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:43.174032 20671 heartbeater.cc:511] Master 127.19.68.62:35091 requested a full tablet report, sending...
I20260810 19:30:43.176132 20337 ts_manager.cc:194] Registered new tserver with Master: 058763074889426c9748d7d110233eaf (127.19.68.1:38367)
I20260810 19:30:43.178025 20337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.1:33053
W20260810 19:30:43.501030 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:43.501360 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:43.501497 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:43.508699 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:43.508801 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:43.508939 20676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:43.517926 20676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:43.519196 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:43.520925 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:43.533517 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.533581 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.533542 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:43.534119 20676 server_base.cc:1034] running on GCE node
I20260810 19:30:43.534715 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:43.535617 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:43.536830 20676 hybrid_clock.cc:648] HybridClock initialized: now 1786390243536772 us; error 52 us; skew 500 ppm
I20260810 19:30:43.537160 20676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:43.539366 20676 webserver.cc:460] Webserver started at http://127.19.68.2:45005/ using document root <none> and password file <none>
I20260810 19:30:43.540038 20676 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:43.540141 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:43.540441 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:43.543407 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "f270be9ae1584bec9f295aba57087fa6"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a86f87d99410ad4f51b59037ecf6914f"
server_key_iv: "740304337be5c30391f9beab9d9cf657"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.544173 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "f270be9ae1584bec9f295aba57087fa6"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a86f87d99410ad4f51b59037ecf6914f"
server_key_iv: "740304337be5c30391f9beab9d9cf657"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.549129 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 19:30:43.552381 20700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.553670 20676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:43.553820 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "f270be9ae1584bec9f295aba57087fa6"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a86f87d99410ad4f51b59037ecf6914f"
server_key_iv: "740304337be5c30391f9beab9d9cf657"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.554023 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:43.570674 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:43.571425 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:43.571671 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:43.572602 20676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:43.574023 20676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:43.574123 20676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.574225 20676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:43.574293 20676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.604619 20676 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:35155
I20260810 19:30:43.604668 20893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:35155 every 8 connection(s)
I20260810 19:30:43.606038 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:43.610374 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20676
I20260810 19:30:43.610556 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:43.611102 19728 external_mini_cluster.cc:1428] Setting key 8245adf3be3a87657b9fba1dc6dcbb65
I20260810 19:30:43.614339 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--builtin_ntp_servers=127.19.68.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:43.619699 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:43.608614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49475 (local address 127.19.68.62:35091)
0810 19:30:43.608803 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:43.608808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:43.610403 (+ 1595us) server_negotiation.cc:407] Connection header received
0810 19:30:43.612565 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:43.612569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:43.612635 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:43.612713 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:43.614052 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.614812 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.616877 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.617295 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.617382 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:43.618801 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:43.618807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:43.618812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:43.618835 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:43.618872 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:43.618888 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:43.619006 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:43.619261 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 19:30:43.619363 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:30:43.620699 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:43.609181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35091 (local address 127.19.68.2:49475)
0810 19:30:43.610218 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:43.610266 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:43.612299 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:43.612975 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:43.612984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:43.613066 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:43.613868 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:43.613895 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.614972 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.614976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:43.616745 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:43.616760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:43.617508 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:43.617515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:43.617770 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:43.618477 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:43.618480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:43.618526 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:43.618611 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:43.619012 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:43.619022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:43.619029 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:30:43.619202 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 19:30:43.619548 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260810 19:30:43.622021 20895 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35091
I20260810 19:30:43.622303 20895 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:43.623145 20895 heartbeater.cc:511] Master 127.19.68.62:35091 requested a full tablet report, sending...
I20260810 19:30:43.624601 20337 ts_manager.cc:194] Registered new tserver with Master: f270be9ae1584bec9f295aba57087fa6 (127.19.68.2:35155)
I20260810 19:30:43.625684 20337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.2:49475
W20260810 19:30:43.945502 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:43.945794 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:43.945909 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:43.953424 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:43.953528 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:43.953663 20900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:43.962993 20900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:43.964303 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:43.966259 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:43.978835 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.978749 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:43.978787 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:43.979687 20900 server_base.cc:1034] running on GCE node
I20260810 19:30:43.980224 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:43.981174 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:43.982376 20900 hybrid_clock.cc:648] HybridClock initialized: now 1786390243982320 us; error 51 us; skew 500 ppm
I20260810 19:30:43.982661 20900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:43.984822 20900 webserver.cc:460] Webserver started at http://127.19.68.3:33931/ using document root <none> and password file <none>
I20260810 19:30:43.985526 20900 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:43.985616 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:43.985872 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:43.988672 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a0c603d5e5714a08bd4e1fd9929c4e77"
server_key_iv: "c836b5fb02d48029086521e0260dbb17"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.989459 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a0c603d5e5714a08bd4e1fd9929c4e77"
server_key_iv: "c836b5fb02d48029086521e0260dbb17"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.994362 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:30:43.997599 20928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:43.998816 20900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:30:43.998962 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f"
format_stamp: "Formatted at 2026-08-10 19:30:43 on dist-test-slave-2j7r"
server_key: "a0c603d5e5714a08bd4e1fd9929c4e77"
server_key_iv: "c836b5fb02d48029086521e0260dbb17"
server_key_version: "encryptionkey@0"
I20260810 19:30:43.999161 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:44.017676 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:44.018430 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:44.018687 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:44.019619 20900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:44.021019 20900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:44.021131 20900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:44.021235 20900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:44.021309 20900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:44.051390 20900 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:44445
I20260810 19:30:44.051466 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:44445 every 8 connection(s)
I20260810 19:30:44.052773 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:44.059245 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 20900
I20260810 19:30:44.059458 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:44.060039 19728 external_mini_cluster.cc:1428] Setting key 8aec29ffcf5b6022976435f3b8b6645d
I20260810 19:30:44.065467 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.055314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:54653 (local address 127.19.68.62:35091)
0810 19:30:44.055543 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.055547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.056896 (+ 1349us) server_negotiation.cc:407] Connection header received
0810 19:30:44.058772 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.058776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.058845 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.058930 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:44.060303 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.061216 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.063112 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.063536 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.063622 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.064821 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:44.064825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:44.064828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:44.064845 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:44.064876 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:44.064887 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:44.064960 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.065147 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.065222 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 19:30:44.066423 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.055779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35091 (local address 127.19.68.3:54653)
0810 19:30:44.056693 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.056734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.058517 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.059199 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.059210 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.059335 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:44.060127 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.060156 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.061403 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.061408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.062988 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.063003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.063723 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.063728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.063926 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.064522 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:44.064525 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:44.064584 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:44.064682 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:44.064966 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:44.064973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:44.064978 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.065143 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.065454 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":295,"threads_started":1}
I20260810 19:30:44.067613 21109 heartbeater.cc:347] Connected to a master server at 127.19.68.62:35091
I20260810 19:30:44.067904 21109 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:44.068599 21109 heartbeater.cc:511] Master 127.19.68.62:35091 requested a full tablet report, sending...
I20260810 19:30:44.069904 20337 ts_manager.cc:194] Registered new tserver with Master: e9a0b2606c3b4a63a2c1c45963c9b16f (127.19.68.3:44445)
I20260810 19:30:44.070710 20337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.3:54653
I20260810 19:30:44.074914 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:44.084714 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.078789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.19.68.62:35091)
0810 19:30:44.079011 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.079017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.079293 (+ 276us) server_negotiation.cc:407] Connection header received
0810 19:30:44.079421 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.079425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.079498 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.079580 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:44.080136 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.080967 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.082330 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.082739 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.082827 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.083960 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:44.083964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:44.083968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:44.083985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:44.084015 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:44.084027 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:44.084138 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.084378 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.084419 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:30:44.088207 20337 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:44.094834 20337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:44.119036 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.107388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:35155 (local address 127.0.0.1:48418)
0810 19:30:44.109608 (+ 2220us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.109652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.109830 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.111828 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.111836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.111873 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:44.112538 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.112552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.114581 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.114587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.116159 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.116172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.118007 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.118014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.118223 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.118279 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.118410 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.118643 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"mutex_wait_us":25,"thread_start_us":3194,"threads_started":1}
I20260810 19:30:44.120096 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.107597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.19.68.2:35155)
0810 19:30:44.108616 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.108623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.109769 (+ 1146us) server_negotiation.cc:407] Connection header received
0810 19:30:44.110119 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.110129 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.111594 (+ 1465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.111793 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:44.112785 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.114422 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.116364 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.117803 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.117932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.118220 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.119551 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.119717 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":612,"threads_started":1}
I20260810 19:30:44.121496 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.108032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.19.68.1:38367)
0810 19:30:44.110440 (+ 2408us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.110456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.111008 (+ 552us) server_negotiation.cc:407] Connection header received
0810 19:30:44.111321 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.111337 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.111591 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.111763 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:44.112872 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.114430 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.118808 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.120129 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.120253 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.120475 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.120893 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.121144 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2240,"thread_start_us":2098,"threads_started":1}
I20260810 19:30:44.121783 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.110713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.19.68.3:44445)
0810 19:30:44.111482 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.111496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.112935 (+ 1439us) server_negotiation.cc:407] Connection header received
0810 19:30:44.113104 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.113113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.113290 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.113444 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:44.114136 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.115434 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.119281 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.120675 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.120809 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.121063 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.121245 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.121423 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":339,"threads_started":1}
I20260810 19:30:44.122252 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.107424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42390)
0810 19:30:44.110896 (+ 3472us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.110913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.111118 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.112061 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.112065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.112088 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:44.112624 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.112642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.114581 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.114587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.118573 (+ 3986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.118588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.120578 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.120582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.120695 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.120717 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.121681 (+ 964us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.121825 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3339,"mutex_wait_us":54,"thread_start_us":1877,"threads_started":1}
I20260810 19:30:44.122344 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.110147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:44445 (local address 127.0.0.1:53100)
0810 19:30:44.112839 (+ 2692us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.112855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.112958 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.113454 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.113459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.113482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:44.113812 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.113821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.115569 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.115574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.118634 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.118652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.120863 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.120869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.121041 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.121075 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.121189 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.121326 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2575,"mutex_wait_us":104,"thread_start_us":2415,"threads_started":1}
I20260810 19:30:44.123880 20777 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb (DEFAULT_TABLE table=test-table [id=23f502c98bc644edaa8113765cf45621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:44.125072 20777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:44.125810 20560 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb (DEFAULT_TABLE table=test-table [id=23f502c98bc644edaa8113765cf45621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:44.125810 20999 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb (DEFAULT_TABLE table=test-table [id=23f502c98bc644edaa8113765cf45621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:44.127159 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:44.127158 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:44.136689 21140 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Bootstrap starting.
I20260810 19:30:44.139756 21140 tablet_bootstrap.cc:654] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:44.141270 21140 log.cc:824] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:44.142629 21146 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Bootstrap starting.
I20260810 19:30:44.144502 21140 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: No bootstrap required, opened a new log
I20260810 19:30:44.144788 21140 ts_tablet_manager.cc:1397] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 19:30:44.145772 21144 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Bootstrap starting.
I20260810 19:30:44.146268 21146 tablet_bootstrap.cc:654] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:44.147432 21146 log.cc:824] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:44.148947 21144 tablet_bootstrap.cc:654] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:44.150142 21144 log.cc:824] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:44.150583 21146 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: No bootstrap required, opened a new log
I20260810 19:30:44.150871 21146 ts_tablet_manager.cc:1397] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 19:30:44.153118 21144 tablet_bootstrap.cc:492] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: No bootstrap required, opened a new log
I20260810 19:30:44.153384 21144 ts_tablet_manager.cc:1397] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260810 19:30:44.153393 21140 raft_consensus.cc:348] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.153760 21140 raft_consensus.cc:374] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:44.153865 21140 raft_consensus.cc:729] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f270be9ae1584bec9f295aba57087fa6, State: Initialized, Role: FOLLOWER
I20260810 19:30:44.154415 21140 consensus_queue.cc:260] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.155426 20895 heartbeater.cc:503] Master 127.19.68.62:35091 was elected leader, sending a full tablet report...
I20260810 19:30:44.156047 21140 ts_tablet_manager.cc:1428] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260810 19:30:44.163259 21146 raft_consensus.cc:348] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.163705 21146 raft_consensus.cc:374] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:44.163825 21146 raft_consensus.cc:729] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0b2606c3b4a63a2c1c45963c9b16f, State: Initialized, Role: FOLLOWER
I20260810 19:30:44.163753 21144 raft_consensus.cc:348] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.164177 21144 raft_consensus.cc:374] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:44.164284 21144 raft_consensus.cc:729] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058763074889426c9748d7d110233eaf, State: Initialized, Role: FOLLOWER
I20260810 19:30:44.164456 21146 consensus_queue.cc:260] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.164855 21144 consensus_queue.cc:260] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.165629 21109 heartbeater.cc:503] Master 127.19.68.62:35091 was elected leader, sending a full tablet report...
I20260810 19:30:44.166231 21146 ts_tablet_manager.cc:1428] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260810 19:30:44.169734 20671 heartbeater.cc:503] Master 127.19.68.62:35091 was elected leader, sending a full tablet report...
I20260810 19:30:44.170363 21144 ts_tablet_manager.cc:1428] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
W20260810 19:30:44.304431 21112 tablet.cc:2369] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:44.308209 21157 raft_consensus.cc:482] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:44.308456 21157 raft_consensus.cc:504] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.310235 21157 leader_election.cc:288] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f270be9ae1584bec9f295aba57087fa6 (127.19.68.2:35155), 058763074889426c9748d7d110233eaf (127.19.68.1:38367)
I20260810 19:30:44.316781 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.310671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:42759 (local address 127.19.68.2:35155)
0810 19:30:44.310958 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.310966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.310989 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:30:44.311071 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.311077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.311153 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.311255 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:44.311706 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.312632 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.314923 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.315947 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.316072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.316183 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.316424 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.316570 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260810 19:30:44.316804 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.310465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:35155 (local address 127.19.68.3:42759)
0810 19:30:44.310734 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.310755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.310881 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.311315 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.311319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.311358 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:44.311563 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.311574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.312783 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.312786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.314764 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.314781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.316119 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.316123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.316270 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.316296 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.316379 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.316481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 19:30:44.317309 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.310677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:36701 (local address 127.19.68.1:38367)
0810 19:30:44.310904 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.310910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.311470 (+ 560us) server_negotiation.cc:407] Connection header received
0810 19:30:44.311607 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.311611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.311695 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.311785 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:44.312232 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.313068 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.315300 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.316407 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.316541 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.316622 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.316970 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.317122 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 19:30:44.317827 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.310539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.19.68.3:36701)
0810 19:30:44.311358 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.311380 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:44.311496 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:44.311816 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:44.311819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:44.311839 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:44.312098 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.312106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.313234 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.313240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.315133 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:44.315144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.316581 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.316588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:44.316767 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.316803 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:30:44.317507 (+ 704us) client_negotiation.cc:240] Negotiation successful
0810 19:30:44.317621 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":500,"threads_started":1}
I20260810 19:30:44.317723 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bc8a9f2e84338a68f11b3b34eeffb" candidate_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270be9ae1584bec9f295aba57087fa6" is_pre_election: true
I20260810 19:30:44.318229 20814 raft_consensus.cc:2454] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a0b2606c3b4a63a2c1c45963c9b16f in term 0.
I20260810 19:30:44.318873 20934 leader_election.cc:302] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9a0b2606c3b4a63a2c1c45963c9b16f, f270be9ae1584bec9f295aba57087fa6; no voters:
I20260810 19:30:44.318744 20595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bc8a9f2e84338a68f11b3b34eeffb" candidate_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058763074889426c9748d7d110233eaf" is_pre_election: true
I20260810 19:30:44.319279 21157 raft_consensus.cc:2781] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:44.319319 20595 raft_consensus.cc:2454] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a0b2606c3b4a63a2c1c45963c9b16f in term 0.
I20260810 19:30:44.319422 21157 raft_consensus.cc:482] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:44.319526 21157 raft_consensus.cc:3037] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:44.321807 21157 raft_consensus.cc:504] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.322381 21157 leader_election.cc:288] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [CANDIDATE]: Term 1 election: Requested vote from peers f270be9ae1584bec9f295aba57087fa6 (127.19.68.2:35155), 058763074889426c9748d7d110233eaf (127.19.68.1:38367)
I20260810 19:30:44.322778 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bc8a9f2e84338a68f11b3b34eeffb" candidate_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270be9ae1584bec9f295aba57087fa6"
I20260810 19:30:44.322865 20595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bc8a9f2e84338a68f11b3b34eeffb" candidate_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058763074889426c9748d7d110233eaf"
I20260810 19:30:44.322969 20814 raft_consensus.cc:3037] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:44.323051 20595 raft_consensus.cc:3037] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:44.325212 20814 raft_consensus.cc:2454] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a0b2606c3b4a63a2c1c45963c9b16f in term 1.
I20260810 19:30:44.325253 20595 raft_consensus.cc:2454] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a0b2606c3b4a63a2c1c45963c9b16f in term 1.
I20260810 19:30:44.325558 20934 leader_election.cc:302] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9a0b2606c3b4a63a2c1c45963c9b16f, f270be9ae1584bec9f295aba57087fa6; no voters:
I20260810 19:30:44.325865 21157 raft_consensus.cc:2781] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:44.326313 21157 raft_consensus.cc:686] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 1 LEADER]: Becoming Leader. State: Replica: e9a0b2606c3b4a63a2c1c45963c9b16f, State: Running, Role: LEADER
I20260810 19:30:44.326821 21157 consensus_queue.cc:237] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } }
I20260810 19:30:44.331583 20336 catalog_manager.cc:5495] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a0b2606c3b4a63a2c1c45963c9b16f (127.19.68.3). New cstate: current_term: 1 leader_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a0b2606c3b4a63a2c1c45963c9b16f" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 44445 } health_report { overall_health: HEALTHY } } }
W20260810 19:30:44.358645 20896 tablet.cc:2369] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:44.359663 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.354303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.19.68.3:44445)
0810 19:30:44.354484 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.354488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.354506 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:30:44.354671 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.354674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.354761 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.354882 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:44.355173 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.355904 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.357478 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.357955 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.358052 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.358574 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:44.359117 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:44.359243 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.359424 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.359476 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:30:44.367790 20814 raft_consensus.cc:1261] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 1 FOLLOWER]: Refusing update from remote peer e9a0b2606c3b4a63a2c1c45963c9b16f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:44.367794 20595 raft_consensus.cc:1261] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 1 FOLLOWER]: Refusing update from remote peer e9a0b2606c3b4a63a2c1c45963c9b16f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:44.368820 21157 consensus_queue.cc:1035] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270be9ae1584bec9f295aba57087fa6" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:44.369136 21161 consensus_queue.cc:1035] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "058763074889426c9748d7d110233eaf" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 38367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:44.383066 21165 mvcc.cc:205] Tried to move back new op lower bound from 7317054440920088576 to 7317054440766558208. Current Snapshot: MvccSnapshot[applied={T|T < 7317054440920088576}]
I20260810 19:30:44.394017 20336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:44.394306 20336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:44.397917 20336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ffd87ef1c15461b915fbfc4c12e1da0: Sending DeleteTablet for 3 replicas of tablet 0d2bc8a9f2e84338a68f11b3b34eeffb
I20260810 19:30:44.399053 20777 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:44 UTC) from {username='slave'} at 127.0.0.1:48418
I20260810 19:30:44.399205 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:44 UTC) from {username='slave'} at 127.0.0.1:42390
I20260810 19:30:44.399617 20999 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:44 UTC) from {username='slave'} at 127.0.0.1:53100
I20260810 19:30:44.399919 21180 tablet_replica.cc:330] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: stopping tablet replica
I20260810 19:30:44.400334 21180 raft_consensus.cc:2229] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:44.400622 21182 tablet_replica.cc:330] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: stopping tablet replica
I20260810 19:30:44.400858 21180 raft_consensus.cc:2258] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:44.401145 21182 raft_consensus.cc:2229] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:30:44.401293 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20456
I20260810 19:30:44.401857 21182 raft_consensus.cc:2258] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:44.404242 21182 ts_tablet_manager.cc:1905] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:44.407383 21180 ts_tablet_manager.cc:1905] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:44.413863 21182 ts_tablet_manager.cc:1918] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:44.414132 21182 log.cc:1196] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/0d2bc8a9f2e84338a68f11b3b34eeffb
I20260810 19:30:44.414757 21182 ts_tablet_manager.cc:1939] T 0d2bc8a9f2e84338a68f11b3b34eeffb P e9a0b2606c3b4a63a2c1c45963c9b16f: Deleting consensus metadata
I20260810 19:30:44.416667 20309 catalog_manager.cc:4841] TS e9a0b2606c3b4a63a2c1c45963c9b16f (127.19.68.3:44445): tablet 0d2bc8a9f2e84338a68f11b3b34eeffb (table test-table [id=23f502c98bc644edaa8113765cf45621]) successfully deleted
I20260810 19:30:44.418795 21181 tablet_replica.cc:330] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: stopping tablet replica
I20260810 19:30:44.419231 21181 raft_consensus.cc:2229] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:44.419713 21181 raft_consensus.cc:2258] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:44.420094 21180 ts_tablet_manager.cc:1918] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:44.420262 21180 log.cc:1196] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/0d2bc8a9f2e84338a68f11b3b34eeffb
I20260810 19:30:44.420854 21180 ts_tablet_manager.cc:1939] T 0d2bc8a9f2e84338a68f11b3b34eeffb P f270be9ae1584bec9f295aba57087fa6: Deleting consensus metadata
I20260810 19:30:44.422854 21181 ts_tablet_manager.cc:1905] T 0d2bc8a9f2e84338a68f11b3b34eeffb P 058763074889426c9748d7d110233eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:44.422888 20313 catalog_manager.cc:4841] TS f270be9ae1584bec9f295aba57087fa6 (127.19.68.2:35155): tablet 0d2bc8a9f2e84338a68f11b3b34eeffb (table test-table [id=23f502c98bc644edaa8113765cf45621]) successfully deleted
I20260810 19:30:44.423825 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.402744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.19.68.1:38367)
0810 19:30:44.403158 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.403165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.413545 (+ 10380us) server_negotiation.cc:407] Connection header received
0810 19:30:44.414019 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.414026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.414137 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.414213 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:44.414910 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.416182 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.418280 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.419031 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.419166 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.421596 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:44.421607 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:44.421625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:44.422425 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:44.422495 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:44.422521 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:44.422637 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.423332 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.423459 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260810 19:30:44.424605 20665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:44.564062 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20456
I20260810 19:30:44.587060 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20676
W20260810 19:30:44.587131 20309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:38367 (error 108)
I20260810 19:30:44.589439 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.588919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42416)
0810 19:30:44.589147 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.589280 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 19:30:44.589704 20309 catalog_manager.cc:4584] TS 058763074889426c9748d7d110233eaf (127.19.68.1:38367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d2bc8a9f2e84338a68f11b3b34eeffb: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
I20260810 19:30:44.595230 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.587833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.19.68.2:35155)
0810 19:30:44.588025 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.588031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.588118 (+ 87us) server_negotiation.cc:407] Connection header received
0810 19:30:44.588451 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.588458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.588590 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.588677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:44.589352 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.590470 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.591731 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.592357 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.592488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.593541 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:44.593553 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:44.593570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:44.594369 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:44.594451 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:44.594479 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:44.594589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.594744 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.594861 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:30:44.596084 20888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:44.643512 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.642945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42432)
0810 19:30:44.643235 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.643338 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 19:30:44.644944 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.644470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42438)
0810 19:30:44.644673 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.644776 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 19:30:44.696966 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20676
I20260810 19:30:44.706218 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.705159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42452)
0810 19:30:44.705869 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.706007 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546}
I20260810 19:30:44.709583 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.709128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42464)
0810 19:30:44.709331 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.709435 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 19:30:44.724984 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20900
I20260810 19:30:44.732450 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.725798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.19.68.3:44445)
0810 19:30:44.726025 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:44.726031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:44.726051 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:30:44.726193 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:44.726200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:44.726298 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:44.726384 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:44.726812 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.727596 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.728846 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:44.729517 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:44.729635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:44.730834 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:44.730848 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:44.730869 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:44.731682 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:44.731744 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:44.731769 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:44.731887 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:44.732010 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 19:30:44.732106 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 19:30:44.733281 21100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:44.790278 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.789751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42476)
0810 19:30:44.789994 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.790101 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 19:30:44.791373 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.790951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42490)
0810 19:30:44.791165 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:44.791254 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 19:30:44.855065 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20900
I20260810 19:30:44.890597 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 20286
I20260810 19:30:44.891960 20404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:45.000521 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 20286
I20260810 19:30:45.000608 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:44.999954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:38367 (local address 127.0.0.1:42498)
0810 19:30:45.000333 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:45.000439 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
2026-08-10T19:30:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2894 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:45 dist-test-slave-2j7r krb5kdc[21201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:45 dist-test-slave-2j7r krb5kdc[21201](info): set up 2 sockets
Aug 10 19:30:45 dist-test-slave-2j7r krb5kdc[21201](info): commencing operation
krb5kdc: starting...
W20260810 19:30:46.277503 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.008s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:46 dist-test-slave-2j7r krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:30:46.595466 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38925
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:46.924654 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:46.924939 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:46.925072 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:46.932423 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:46.932519 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:46.932593 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:46.932668 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:46.932725 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:46.942617 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:38925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:46.943921 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:46.945261 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:46.956884 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:46.956935 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:46.956974 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:46.957772 21229 server_base.cc:1034] running on GCE node
I20260810 19:30:46.958564 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:46.959945 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:46.961184 21229 hybrid_clock.cc:648] HybridClock initialized: now 1786390246961129 us; error 40 us; skew 500 ppm
I20260810 19:30:46.961542 21229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:46 dist-test-slave-2j7r krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:46.964576 21229 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:30:46.966082 21229 webserver.cc:460] Webserver started at http://127.19.68.62:37049/ using document root <none> and password file <none>
I20260810 19:30:46.966717 21229 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:46.966816 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:46.967079 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:46.970052 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d"
format_stamp: "Formatted at 2026-08-10 19:30:46 on dist-test-slave-2j7r"
server_key: "aaa955b8f7beae91eb964fe928f1073a"
server_key_iv: "be3e6593bd0a194cca14742ef1b434c5"
server_key_version: "encryptionkey@0"
I20260810 19:30:46.970738 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d"
format_stamp: "Formatted at 2026-08-10 19:30:46 on dist-test-slave-2j7r"
server_key: "aaa955b8f7beae91eb964fe928f1073a"
server_key_iv: "be3e6593bd0a194cca14742ef1b434c5"
server_key_version: "encryptionkey@0"
I20260810 19:30:46.975769 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:30:46.979110 21250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:46.980371 21229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:30:46.980516 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d"
format_stamp: "Formatted at 2026-08-10 19:30:46 on dist-test-slave-2j7r"
server_key: "aaa955b8f7beae91eb964fe928f1073a"
server_key_iv: "be3e6593bd0a194cca14742ef1b434c5"
server_key_version: "encryptionkey@0"
I20260810 19:30:46.980690 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:46.992205 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:46.995879 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:46.996120 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:47.013304 21229 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:38925
I20260810 19:30:47.013338 21340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:38925 every 8 connection(s)
I20260810 19:30:47.014679 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:47.018404 21341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:47.022679 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21229
I20260810 19:30:47.022882 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:47.023406 19728 external_mini_cluster.cc:1428] Setting key 80837f92dd9484bbc1bc65c302db2d10
I20260810 19:30:47.029675 21341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: Bootstrap starting.
I20260810 19:30:47.033195 21341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:47.034332 21341 log.cc:824] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:47.036924 21341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: No bootstrap required, opened a new log
Aug 10 19:30:47 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:47.044647 21341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } }
I20260810 19:30:47.045019 21341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:47.045166 21341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1c23c855274b42b0b7fbf04a7bcd3d, State: Initialized, Role: FOLLOWER
I20260810 19:30:47.045717 21341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } }
I20260810 19:30:47.045933 21341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:47.046084 21341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:47.046275 21341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:47.046279 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:47.025353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.19.68.62:38925)
0810 19:30:47.026129 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:47.026144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:47.026189 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:30:47.028176 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:47.028197 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:47.028557 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:47.028851 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:47.030698 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.032011 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.033826 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.034515 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.034647 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:47.038892 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:47.038903 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:47.038916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:47.038950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:47.042085 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:47.042768 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:47.042774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:47.042780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:47.042983 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:47.043364 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:47.043371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:47.043373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:47.043904 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:47.044175 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:47.044618 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 19:30:47.045073 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":351,"threads_started":1}
I20260810 19:30:47.048632 21341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } }
I20260810 19:30:47.049080 21341 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1c23c855274b42b0b7fbf04a7bcd3d; no voters:
I20260810 19:30:47.049717 21341 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:47.049840 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:47.050472 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [term 1 LEADER]: Becoming Leader. State: Replica: 6d1c23c855274b42b0b7fbf04a7bcd3d, State: Running, Role: LEADER
I20260810 19:30:47.050959 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } }
I20260810 19:30:47.051590 21341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:47.053953 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1c23c855274b42b0b7fbf04a7bcd3d. Latest consensus state: current_term: 1 leader_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } } }
I20260810 19:30:47.054363 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:47.056313 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c23c855274b42b0b7fbf04a7bcd3d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 38925 } } }
I20260810 19:30:47.056702 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:47.057585 21365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:47.063265 21365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:47.072144 21365 catalog_manager.cc:1347] Generated new cluster ID: 0b830dd559cd408bbeb00da8f9ccd2d4
I20260810 19:30:47.072253 21365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:47.097208 21365 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:47.099136 21365 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:47.111991 21365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: Generated new TSK 0
I20260810 19:30:47.112864 21365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
Entry for principal kudu/127.19.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab.
I20260810 19:30:47.192832 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:47.544642 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:47.545076 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:47.545269 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:47.555882 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:47.555987 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:47.556152 21379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:47.566114 21379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.1.keytab
--principal=kudu/127.19.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:47.567433 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:47.568742 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:47.581374 21389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:47.581413 21393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:47.581441 21395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:47.582122 21379 server_base.cc:1034] running on GCE node
I20260810 19:30:47.582715 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:47.583587 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:47.584779 21379 hybrid_clock.cc:648] HybridClock initialized: now 1786390247584738 us; error 53 us; skew 500 ppm
I20260810 19:30:47.585122 21379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:47 dist-test-slave-2j7r krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390247, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:47.588057 21379 init.cc:377] Logged in from keytab as kudu/127.19.68.1@KRBTEST.COM (short username kudu)
I20260810 19:30:47.589506 21379 webserver.cc:460] Webserver started at http://127.19.68.1:35809/ using document root <none> and password file <none>
I20260810 19:30:47.590134 21379 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:47.590255 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:47.590553 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:47.593374 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "fc7d9afac76044cba9e3e17689a2b537"
format_stamp: "Formatted at 2026-08-10 19:30:47 on dist-test-slave-2j7r"
server_key: "fd37816b09460e1554517a36074a3263"
server_key_iv: "49fa7ab043d0ffe790e3632801ce415e"
server_key_version: "encryptionkey@0"
I20260810 19:30:47.594101 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "fc7d9afac76044cba9e3e17689a2b537"
format_stamp: "Formatted at 2026-08-10 19:30:47 on dist-test-slave-2j7r"
server_key: "fd37816b09460e1554517a36074a3263"
server_key_iv: "49fa7ab043d0ffe790e3632801ce415e"
server_key_version: "encryptionkey@0"
I20260810 19:30:47.598914 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 19:30:47.602136 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:47.603330 21379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:30:47.603479 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "fc7d9afac76044cba9e3e17689a2b537"
format_stamp: "Formatted at 2026-08-10 19:30:47 on dist-test-slave-2j7r"
server_key: "fd37816b09460e1554517a36074a3263"
server_key_iv: "49fa7ab043d0ffe790e3632801ce415e"
server_key_version: "encryptionkey@0"
I20260810 19:30:47.603650 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:47.623965 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:47.627584 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:47.627844 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:47.628772 21379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:47.630085 21379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:47.630168 21379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:47.630295 21379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:47.630355 21379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:47.662614 21379 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:40393
I20260810 19:30:47.662643 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:40393 every 8 connection(s)
I20260810 19:30:47.663976 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:47.664625 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21379
I20260810 19:30:47.664799 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:47.665410 19728 external_mini_cluster.cc:1428] Setting key d71dab41236c243f7e7b501c2d601849
Aug 10 19:30:47 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390247, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.1@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:47.684914 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:47.666722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:40163 (local address 127.19.68.62:38925)
0810 19:30:47.667220 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:47.667227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:47.668413 (+ 1186us) server_negotiation.cc:407] Connection header received
0810 19:30:47.670320 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:47.670327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:47.670425 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:47.670551 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:47.671981 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.672977 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.674492 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.675083 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.675240 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:47.678733 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:47.678738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:47.678742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:47.678773 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:47.681228 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:47.682142 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:47.682147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:47.682150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:47.682305 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:47.682723 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:47.682728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:47.682731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:47.683086 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:47.683252 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:47.684152 (+ 900us) server_negotiation.cc:299] Negotiation successful
0810 19:30:47.684441 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260810 19:30:47.685804 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:47.667366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38925 (local address 127.19.68.1:40163)
0810 19:30:47.668233 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:47.668277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:47.670052 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:47.670727 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:47.670735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:47.671231 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:47.671801 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:47.671819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.673142 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.673145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:47.674343 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:47.674356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:47.675293 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:47.675297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:47.675499 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:47.676109 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:47.676136 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:47.678507 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:47.681369 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:47.681378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:47.681394 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:47.681998 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:47.682404 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:47.682408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:47.682410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:47.682612 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:47.683243 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:47.683252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:47.683631 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 19:30:47.683932 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 19:30:47.684491 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":333,"threads_started":1}
I20260810 19:30:47.687430 21603 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38925
I20260810 19:30:47.687759 21603 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:47.688531 21603 heartbeater.cc:511] Master 127.19.68.62:38925 requested a full tablet report, sending...
I20260810 19:30:47.690718 21276 ts_manager.cc:194] Registered new tserver with Master: fc7d9afac76044cba9e3e17689a2b537 (127.19.68.1:40393)
I20260810 19:30:47.692234 21276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.1@KRBTEST.COM'} at 127.19.68.1:40163
WARNING: no policy specified for kudu/127.19.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
Entry for principal kudu/127.19.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab.
I20260810 19:30:47.734341 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:48.075045 21616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:48.075350 21616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:48.075460 21616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:48.082778 21616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:48.082890 21616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:48.083017 21616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:48.093252 21616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.2.keytab
--principal=kudu/127.19.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:48.094563 21616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:48.095914 21616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:48.109501 21628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:48.109540 21626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:48.109655 21625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:48.110559 21616 server_base.cc:1034] running on GCE node
I20260810 19:30:48.111120 21616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:48.111991 21616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:48.113181 21616 hybrid_clock.cc:648] HybridClock initialized: now 1786390248113119 us; error 52 us; skew 500 ppm
I20260810 19:30:48.113456 21616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:48 dist-test-slave-2j7r krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390248, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:48.119063 21616 init.cc:377] Logged in from keytab as kudu/127.19.68.2@KRBTEST.COM (short username kudu)
I20260810 19:30:48.120406 21616 webserver.cc:460] Webserver started at http://127.19.68.2:34979/ using document root <none> and password file <none>
I20260810 19:30:48.121040 21616 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:48.121186 21616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:48.121466 21616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:48.124256 21616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "f5df15a4838c4f368bfd7454fa389690"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "3282c455f0032266f532663c51010c39"
server_key_iv: "424fb0b869e4897761a6b9d92f689b9a"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.124969 21616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "f5df15a4838c4f368bfd7454fa389690"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "3282c455f0032266f532663c51010c39"
server_key_iv: "424fb0b869e4897761a6b9d92f689b9a"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.130282 21616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:30:48.133679 21641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.135099 21616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:30:48.135239 21616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "f5df15a4838c4f368bfd7454fa389690"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "3282c455f0032266f532663c51010c39"
server_key_iv: "424fb0b869e4897761a6b9d92f689b9a"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.135419 21616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:48.147730 21616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:48.151850 21616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:48.152105 21616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:48.152987 21616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:48.154379 21616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:48.154450 21616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.154539 21616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:48.154582 21616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.186949 21616 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:46743
I20260810 19:30:48.186987 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:46743 every 8 connection(s)
I20260810 19:30:48.188201 21616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:48.194597 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21616
I20260810 19:30:48.194766 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:48.195179 19728 external_mini_cluster.cc:1428] Setting key 18a8ee7fda29084cdf184c167b2b2613
Aug 10 19:30:48 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390248, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.2@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:48.208230 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.190676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:43673 (local address 127.19.68.62:38925)
0810 19:30:48.191153 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.191159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.192011 (+ 852us) server_negotiation.cc:407] Connection header received
0810 19:30:48.193848 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.193853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.193930 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.194012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:48.195533 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.196402 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.198222 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.198770 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.198917 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.202619 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:48.202625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:48.202628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:48.202664 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:48.205133 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.205878 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.205882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.205884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.206040 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.206484 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.206488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.206490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.206758 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:48.206895 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.207611 (+ 716us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.207844 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260810 19:30:48.209214 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.191137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38925 (local address 127.19.68.2:43673)
0810 19:30:48.191845 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.191883 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.193579 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.194204 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.194212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.194713 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:48.195357 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.195376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.196554 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.196559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.198078 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.198091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.199015 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.199021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.199225 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.199848 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:48.199871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:48.202338 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:48.205285 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:48.205295 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:48.205308 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:48.205747 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:48.206157 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:48.206162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:48.206166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:48.206360 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:48.206886 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:48.206894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:48.207229 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.207476 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.208019 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":266,"threads_started":1}
I20260810 19:30:48.210625 21830 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38925
I20260810 19:30:48.210922 21830 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:48.211633 21830 heartbeater.cc:511] Master 127.19.68.62:38925 requested a full tablet report, sending...
I20260810 19:30:48.213186 21276 ts_manager.cc:194] Registered new tserver with Master: f5df15a4838c4f368bfd7454fa389690 (127.19.68.2:46743)
I20260810 19:30:48.213999 21276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.2@KRBTEST.COM'} at 127.19.68.2:43673
WARNING: no policy specified for kudu/127.19.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
Entry for principal kudu/127.19.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab.
I20260810 19:30:48.265309 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:48.600131 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:48.600445 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:48.600566 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:48.607666 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:48.607765 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:48.607906 21838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:48.617306 21838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.3.keytab
--principal=kudu/127.19.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:48.618474 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:48.619678 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:48.632189 21847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:48.632269 21851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:48.632159 21846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:48.632871 21838 server_base.cc:1034] running on GCE node
I20260810 19:30:48.633551 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:48.634533 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:48.635731 21838 hybrid_clock.cc:648] HybridClock initialized: now 1786390248635676 us; error 46 us; skew 500 ppm
I20260810 19:30:48.636065 21838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:48 dist-test-slave-2j7r krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390248, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:48.639164 21838 init.cc:377] Logged in from keytab as kudu/127.19.68.3@KRBTEST.COM (short username kudu)
I20260810 19:30:48.640578 21838 webserver.cc:460] Webserver started at http://127.19.68.3:33217/ using document root <none> and password file <none>
I20260810 19:30:48.641275 21838 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:48.641402 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:48.641695 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:48.644599 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "a0a6142278a5439e9d0ac705a6c3a7ed"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "e4b7df9971e577f5b635ecbc1f0f8a05"
server_key_iv: "b35562928c8affe0909f1d0fc4a78691"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.645392 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "a0a6142278a5439e9d0ac705a6c3a7ed"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "e4b7df9971e577f5b635ecbc1f0f8a05"
server_key_iv: "b35562928c8affe0909f1d0fc4a78691"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.650311 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:30:48.653708 21860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.654951 21838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:30:48.655082 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "a0a6142278a5439e9d0ac705a6c3a7ed"
format_stamp: "Formatted at 2026-08-10 19:30:48 on dist-test-slave-2j7r"
server_key: "e4b7df9971e577f5b635ecbc1f0f8a05"
server_key_iv: "b35562928c8affe0909f1d0fc4a78691"
server_key_version: "encryptionkey@0"
I20260810 19:30:48.655232 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:48.683964 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:48.687832 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:48.688078 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:48.688979 21838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:48.690333 21838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:48.690407 21838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.690495 21838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:48.690546 21838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:48.695288 21603 heartbeater.cc:503] Master 127.19.68.62:38925 was elected leader, sending a full tablet report...
I20260810 19:30:48.720920 21838 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:39877
I20260810 19:30:48.720978 22044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:39877 every 8 connection(s)
I20260810 19:30:48.722227 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:48.725788 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 21838
I20260810 19:30:48.726006 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:48.726523 19728 external_mini_cluster.cc:1428] Setting key ce9df5b35bcf5ddf9c1fc6963525a02f
Aug 10 19:30:48 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390248, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.3@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:30:48.741959 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.724628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:33469 (local address 127.19.68.62:38925)
0810 19:30:48.725045 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.725050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.726119 (+ 1069us) server_negotiation.cc:407] Connection header received
0810 19:30:48.728207 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.728212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.728310 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.728446 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:48.730248 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.731089 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.732581 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.733009 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.733171 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.736370 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:48.736375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:48.736379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:48.736424 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:48.738832 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.739552 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.739556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.739558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.739698 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.740178 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.740181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.740183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.740466 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:48.740613 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.741371 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.741597 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":195,"threads_started":1}
I20260810 19:30:48.742897 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.725177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:38925 (local address 127.19.68.3:33469)
0810 19:30:48.725874 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.725922 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.727944 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.728697 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.728708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.729489 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:48.730046 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.730063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.731222 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.731225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.732430 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.732443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.733225 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.733230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.733428 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.734022 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:30:48.734044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:48.736160 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:48.738999 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:48.739005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:48.739017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:48.739452 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:48.739809 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:30:48.739814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:30:48.739818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:30:48.740060 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:30:48.740598 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:48.740606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:48.740940 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.741213 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.741745 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":301,"threads_started":1}
I20260810 19:30:48.744110 22045 heartbeater.cc:347] Connected to a master server at 127.19.68.62:38925
I20260810 19:30:48.744395 22045 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:48.745115 22045 heartbeater.cc:511] Master 127.19.68.62:38925 requested a full tablet report, sending...
I20260810 19:30:48.746450 21276 ts_manager.cc:194] Registered new tserver with Master: a0a6142278a5439e9d0ac705a6c3a7ed (127.19.68.3:39877)
I20260810 19:30:48.747238 21276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.68.3@KRBTEST.COM'} at 127.19.68.3:33469
I20260810 19:30:48.753647 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:48.769191 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.757638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.19.68.62:38925)
0810 19:30:48.757842 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.757849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.758252 (+ 403us) server_negotiation.cc:407] Connection header received
0810 19:30:48.758385 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.758389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.758489 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.758567 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:48.760002 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.760701 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.761960 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.762382 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.762506 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.764638 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:48.764643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:48.764646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:48.764677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:48.767018 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.767580 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.767583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.767586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.767691 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:48.767980 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:48.767983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:48.767985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:48.768317 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:48.768505 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.768747 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.768939 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:30:48.772614 21276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:48.779301 21276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:48.803334 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.791049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46743 (local address 127.0.0.1:41726)
0810 19:30:48.792856 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.792891 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.793059 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.795884 (+ 2825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.795888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.795919 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:48.796486 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.796499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.798052 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.798057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.800561 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.800576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802306 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.802313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.802503 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.802561 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.802670 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.802865 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1592,"mutex_wait_us":49,"thread_start_us":2500,"threads_started":1}
I20260810 19:30:48.803399 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.793592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.19.68.1:40393)
0810 19:30:48.794203 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.794212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.794789 (+ 577us) server_negotiation.cc:407] Connection header received
0810 19:30:48.795096 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.795113 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.795337 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.795458 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:48.796639 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.798243 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.800563 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802025 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.802152 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.802413 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.802898 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.803079 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
I20260810 19:30:48.803928 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.791052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.19.68.2:46743)
0810 19:30:48.795369 (+ 4317us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.795378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.795403 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:30:48.795490 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.795497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.795698 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.795884 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:48.796669 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.797877 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.800718 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802106 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.803250 (+ 1144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.803450 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.803520 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.803685 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4140,"thread_start_us":4007,"threads_started":1}
I20260810 19:30:48.803934 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.791497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.19.68.3:39877)
0810 19:30:48.792346 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.792356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.794647 (+ 2291us) server_negotiation.cc:407] Connection header received
0810 19:30:48.794968 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.794976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.795238 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.795402 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:48.796513 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.798251 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.800977 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802167 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.802292 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.802519 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.802992 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.803142 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":484,"threads_started":1}
I20260810 19:30:48.804286 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.793243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:40393 (local address 127.0.0.1:45132)
0810 19:30:48.794672 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.794691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.794897 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.795707 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.795711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.795733 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:48.796427 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.796444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.798413 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.798419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.800413 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.800426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802550 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.802556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.802715 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.802744 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.802823 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.802922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":138,"thread_start_us":1132,"threads_started":1}
I20260810 19:30:48.804613 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.791111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:39877 (local address 127.0.0.1:59310)
0810 19:30:48.793847 (+ 2736us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.794557 (+ 710us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.794757 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.795740 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.795745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.795769 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:48.796332 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.796345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.798413 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.798418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.800834 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.800844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.802655 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.802660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.802784 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.802813 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.804182 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.804295 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2609,"mutex_wait_us":109,"thread_start_us":1397,"threads_started":1}
I20260810 19:30:48.807766 21720 tablet_service.cc:1467] Processing CreateTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 (DEFAULT_TABLE table=test-table [id=e045953f10804212bbfdd295c4755fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:48.807854 21490 tablet_service.cc:1467] Processing CreateTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 (DEFAULT_TABLE table=test-table [id=e045953f10804212bbfdd295c4755fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:48.808466 21934 tablet_service.cc:1467] Processing CreateTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 (DEFAULT_TABLE table=test-table [id=e045953f10804212bbfdd295c4755fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:48.808984 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb75271253e54092ab1ad2eeafdac1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:48.809271 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb75271253e54092ab1ad2eeafdac1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:48.809804 21934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb75271253e54092ab1ad2eeafdac1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:48.820855 22073 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Bootstrap starting.
I20260810 19:30:48.823563 22074 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Bootstrap starting.
I20260810 19:30:48.823760 22073 tablet_bootstrap.cc:654] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:48.824393 22075 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Bootstrap starting.
I20260810 19:30:48.825040 22073 log.cc:824] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:48.827276 22074 tablet_bootstrap.cc:654] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:48.827767 22075 tablet_bootstrap.cc:654] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:48.828317 22074 log.cc:824] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:48.828971 22075 log.cc:824] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:48.829437 22073 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: No bootstrap required, opened a new log
I20260810 19:30:48.829716 22073 ts_tablet_manager.cc:1397] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 19:30:48.831904 22075 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: No bootstrap required, opened a new log
I20260810 19:30:48.832170 22075 ts_tablet_manager.cc:1397] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:30:48.832237 22074 tablet_bootstrap.cc:492] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: No bootstrap required, opened a new log
I20260810 19:30:48.832470 22074 ts_tablet_manager.cc:1397] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 19:30:48.836686 22073 raft_consensus.cc:348] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.837023 22073 raft_consensus.cc:374] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:48.837141 22073 raft_consensus.cc:729] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5df15a4838c4f368bfd7454fa389690, State: Initialized, Role: FOLLOWER
I20260810 19:30:48.837693 22073 consensus_queue.cc:260] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.839766 21830 heartbeater.cc:503] Master 127.19.68.62:38925 was elected leader, sending a full tablet report...
I20260810 19:30:48.840363 22073 ts_tablet_manager.cc:1428] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260810 19:30:48.842213 22074 raft_consensus.cc:348] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.842520 22074 raft_consensus.cc:374] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:48.842613 22074 raft_consensus.cc:729] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a6142278a5439e9d0ac705a6c3a7ed, State: Initialized, Role: FOLLOWER
I20260810 19:30:48.843158 22074 consensus_queue.cc:260] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.843681 22075 raft_consensus.cc:348] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.844081 22075 raft_consensus.cc:374] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:48.844174 22075 raft_consensus.cc:729] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7d9afac76044cba9e3e17689a2b537, State: Initialized, Role: FOLLOWER
I20260810 19:30:48.844714 22075 consensus_queue.cc:260] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.845129 22045 heartbeater.cc:503] Master 127.19.68.62:38925 was elected leader, sending a full tablet report...
I20260810 19:30:48.845810 22074 ts_tablet_manager.cc:1428] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260810 19:30:48.850354 22075 ts_tablet_manager.cc:1428] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
W20260810 19:30:48.916550 21605 tablet.cc:2369] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:30:48.940127 21831 tablet.cc:2369] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:48.966315 22085 raft_consensus.cc:482] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:48.966635 22085 raft_consensus.cc:504] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.968313 22085 leader_election.cc:288] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5df15a4838c4f368bfd7454fa389690 (127.19.68.2:46743), a0a6142278a5439e9d0ac705a6c3a7ed (127.19.68.3:39877)
W20260810 19:30:48.973718 22047 tablet.cc:2369] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:30:48.976912 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.968858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:56479 (local address 127.19.68.3:39877)
0810 19:30:48.969235 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.969242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.969604 (+ 362us) server_negotiation.cc:407] Connection header received
0810 19:30:48.969811 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.969815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.969904 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.969992 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:48.970681 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.971813 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.974664 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.975881 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.976003 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.976105 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.976529 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.976657 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260810 19:30:48.976912 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.968811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:36321 (local address 127.19.68.2:46743)
0810 19:30:48.969023 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:48.969030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:48.969869 (+ 839us) server_negotiation.cc:407] Connection header received
0810 19:30:48.970003 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:48.970009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:48.970081 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:48.970167 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:48.970818 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.971843 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.974687 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.975870 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.976008 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.976098 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:48.976528 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 19:30:48.976657 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:30:48.977157 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.968710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:46743 (local address 127.19.68.1:36321)
0810 19:30:48.969766 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.969780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.969891 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.970206 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.970211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.970235 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:48.970680 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.970692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.971956 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.971961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.974511 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.974527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.976135 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.976141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.976350 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.976403 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.976492 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.976648 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":199,"thread_start_us":747,"threads_started":1}
I20260810 19:30:48.977180 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:48.968614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:39877 (local address 127.19.68.1:56479)
0810 19:30:48.969489 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:48.969505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:48.969674 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:48.970036 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:48.970039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:48.970062 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:48.970522 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.970537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.971956 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.971961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.974523 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:48.974532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:48.976097 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:48.976104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:48.976352 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:48.976398 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 19:30:48.976491 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:30:48.976648 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":240,"thread_start_us":554,"threads_started":1}
I20260810 19:30:48.978264 21965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb75271253e54092ab1ad2eeafdac1f8" candidate_uuid: "fc7d9afac76044cba9e3e17689a2b537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" is_pre_election: true
I20260810 19:30:48.978250 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb75271253e54092ab1ad2eeafdac1f8" candidate_uuid: "fc7d9afac76044cba9e3e17689a2b537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5df15a4838c4f368bfd7454fa389690" is_pre_election: true
I20260810 19:30:48.978827 21752 raft_consensus.cc:2454] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7d9afac76044cba9e3e17689a2b537 in term 0.
I20260810 19:30:48.978823 21965 raft_consensus.cc:2454] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7d9afac76044cba9e3e17689a2b537 in term 0.
I20260810 19:30:48.979465 21408 leader_election.cc:302] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a6142278a5439e9d0ac705a6c3a7ed, fc7d9afac76044cba9e3e17689a2b537; no voters:
I20260810 19:30:48.979828 22085 raft_consensus.cc:2781] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:48.979988 22085 raft_consensus.cc:482] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:48.980096 22085 raft_consensus.cc:3037] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:48.982246 22085 raft_consensus.cc:504] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.982795 22085 leader_election.cc:288] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [CANDIDATE]: Term 1 election: Requested vote from peers f5df15a4838c4f368bfd7454fa389690 (127.19.68.2:46743), a0a6142278a5439e9d0ac705a6c3a7ed (127.19.68.3:39877)
I20260810 19:30:48.983143 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb75271253e54092ab1ad2eeafdac1f8" candidate_uuid: "fc7d9afac76044cba9e3e17689a2b537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5df15a4838c4f368bfd7454fa389690"
I20260810 19:30:48.983263 21965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb75271253e54092ab1ad2eeafdac1f8" candidate_uuid: "fc7d9afac76044cba9e3e17689a2b537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed"
I20260810 19:30:48.983318 21752 raft_consensus.cc:3037] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:48.983453 21965 raft_consensus.cc:3037] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:48.985662 21752 raft_consensus.cc:2454] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7d9afac76044cba9e3e17689a2b537 in term 1.
I20260810 19:30:48.985675 21965 raft_consensus.cc:2454] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7d9afac76044cba9e3e17689a2b537 in term 1.
I20260810 19:30:48.986033 21408 leader_election.cc:302] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a6142278a5439e9d0ac705a6c3a7ed, fc7d9afac76044cba9e3e17689a2b537; no voters:
I20260810 19:30:48.986342 22085 raft_consensus.cc:2781] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:48.986737 22085 raft_consensus.cc:686] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 1 LEADER]: Becoming Leader. State: Replica: fc7d9afac76044cba9e3e17689a2b537, State: Running, Role: LEADER
I20260810 19:30:48.987208 22085 consensus_queue.cc:237] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } }
I20260810 19:30:48.991367 21276 catalog_manager.cc:5495] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc7d9afac76044cba9e3e17689a2b537 (127.19.68.1). New cstate: current_term: 1 leader_uuid: "fc7d9afac76044cba9e3e17689a2b537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc7d9afac76044cba9e3e17689a2b537" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 40393 } health_report { overall_health: HEALTHY } } }
I20260810 19:30:49.035506 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:49.030804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.19.68.1:40393)
0810 19:30:49.030977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:49.030982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:49.030999 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:30:49.031125 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:49.031129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:49.031206 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:49.031322 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:49.031630 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.032344 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.033680 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.034072 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.034160 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:49.034566 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:49.035060 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:49.035165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:49.035228 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 19:30:49.035293 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:30:49.043592 21965 raft_consensus.cc:1261] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 1 FOLLOWER]: Refusing update from remote peer fc7d9afac76044cba9e3e17689a2b537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:49.043609 21752 raft_consensus.cc:1261] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 1 FOLLOWER]: Refusing update from remote peer fc7d9afac76044cba9e3e17689a2b537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:49.044507 22085 consensus_queue.cc:1035] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a6142278a5439e9d0ac705a6c3a7ed" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:49.044786 22091 consensus_queue.cc:1035] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5df15a4838c4f368bfd7454fa389690" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:49.071979 21276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:49.072217 21276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:49.075836 21276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d1c23c855274b42b0b7fbf04a7bcd3d: Sending DeleteTablet for 3 replicas of tablet fb75271253e54092ab1ad2eeafdac1f8
I20260810 19:30:49.077075 21720 tablet_service.cc:1514] Processing DeleteTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:41726
I20260810 19:30:49.077144 21934 tablet_service.cc:1514] Processing DeleteTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:59310
I20260810 19:30:49.077514 21490 tablet_service.cc:1514] Processing DeleteTablet for tablet fb75271253e54092ab1ad2eeafdac1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:49 UTC) from {username='kudu', principal='kudu/127.19.68.62@KRBTEST.COM'} at 127.0.0.1:45132
I20260810 19:30:49.078630 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21379
I20260810 19:30:49.089536 22113 tablet_replica.cc:330] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: stopping tablet replica
I20260810 19:30:49.090022 22113 raft_consensus.cc:2229] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:30:49.090736 22113 raft_consensus.cc:2258] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:49.095363 22113 ts_tablet_manager.cc:1905] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:49.098475 22110 tablet_replica.cc:330] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: stopping tablet replica
I20260810 19:30:49.098953 22110 raft_consensus.cc:2229] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:49.099489 22110 raft_consensus.cc:2258] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:49.100268 22112 tablet_replica.cc:330] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: stopping tablet replica
I20260810 19:30:49.100669 22112 raft_consensus.cc:2229] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:49.101156 22112 raft_consensus.cc:2258] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:49.103457 22112 ts_tablet_manager.cc:1905] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:49.106456 22113 ts_tablet_manager.cc:1918] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:49.106635 22113 log.cc:1196] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-0/wal/wals/fb75271253e54092ab1ad2eeafdac1f8
Aug 10 19:30:49 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.1@KRBTEST.COM
I20260810 19:30:49.107266 22113 ts_tablet_manager.cc:1939] T fb75271253e54092ab1ad2eeafdac1f8 P fc7d9afac76044cba9e3e17689a2b537: Deleting consensus metadata
I20260810 19:30:49.109274 21251 catalog_manager.cc:4841] TS fc7d9afac76044cba9e3e17689a2b537 (127.19.68.1:40393): tablet fb75271253e54092ab1ad2eeafdac1f8 (table test-table [id=e045953f10804212bbfdd295c4755fad]) successfully deleted
I20260810 19:30:49.110427 22112 ts_tablet_manager.cc:1918] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:49.110597 22112 log.cc:1196] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/fb75271253e54092ab1ad2eeafdac1f8
I20260810 19:30:49.111152 22112 ts_tablet_manager.cc:1939] T fb75271253e54092ab1ad2eeafdac1f8 P f5df15a4838c4f368bfd7454fa389690: Deleting consensus metadata
I20260810 19:30:49.112267 22110 ts_tablet_manager.cc:1905] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:49.112922 21255 catalog_manager.cc:4841] TS f5df15a4838c4f368bfd7454fa389690 (127.19.68.2:46743): tablet fb75271253e54092ab1ad2eeafdac1f8 (table test-table [id=e045953f10804212bbfdd295c4755fad]) successfully deleted
I20260810 19:30:49.116034 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:49.079825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.19.68.1:40393)
0810 19:30:49.079996 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:49.080003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:49.095151 (+ 15148us) server_negotiation.cc:407] Connection header received
0810 19:30:49.095620 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:49.095627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:49.095707 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:49.095824 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:49.097769 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.099016 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.100892 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.101615 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.101727 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:49.108153 (+ 6426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:49.108169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:49.108188 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:49.108240 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:49.112692 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.113638 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.113648 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.113652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.113827 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.114277 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.114283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.114286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.114751 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:49.115059 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:49.115405 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 19:30:49.115658 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:30:49.116883 21596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:49.118942 22110 ts_tablet_manager.cc:1918] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:49.119159 22110 log.cc:1196] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/fb75271253e54092ab1ad2eeafdac1f8
I20260810 19:30:49.119824 22110 ts_tablet_manager.cc:1939] T fb75271253e54092ab1ad2eeafdac1f8 P a0a6142278a5439e9d0ac705a6c3a7ed: Deleting consensus metadata
I20260810 19:30:49.121532 21251 catalog_manager.cc:4841] TS a0a6142278a5439e9d0ac705a6c3a7ed (127.19.68.3:39877): tablet fb75271253e54092ab1ad2eeafdac1f8 (table test-table [id=e045953f10804212bbfdd295c4755fad]) successfully deleted
I20260810 19:30:49.226138 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21379
I20260810 19:30:49.251302 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21616
Aug 10 19:30:49 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.2@KRBTEST.COM
I20260810 19:30:49.266042 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:49.252132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.19.68.2:46743)
0810 19:30:49.252342 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:49.252349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:49.252429 (+ 80us) server_negotiation.cc:407] Connection header received
0810 19:30:49.252582 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:49.252586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:49.252665 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:49.252784 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:49.254032 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.254742 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.256050 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.256436 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.256536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:49.260720 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:49.260736 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:49.260751 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:49.260785 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:49.263592 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.264223 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.264230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.264233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.264348 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.264642 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.264646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.264647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.265024 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:49.265305 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:49.265601 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 19:30:49.265765 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:30:49.266764 21819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:49.357053 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21616
I20260810 19:30:49.386818 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21838
Aug 10 19:30:49 dist-test-slave-2j7r krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.3@KRBTEST.COM
I20260810 19:30:49.402263 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:49.387565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.19.68.3:39877)
0810 19:30:49.387745 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:49.387751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:49.387861 (+ 110us) server_negotiation.cc:407] Connection header received
0810 19:30:49.388030 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:49.388034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:49.388112 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:49.388200 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:49.389712 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.390453 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.391688 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:49.392117 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:49.392224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:49.396577 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:49.396594 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:49.396609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:49.396646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:49.399541 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.400183 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.400191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.400193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.400358 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:49.400725 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:49.400729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:49.400731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:49.401118 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:49.401431 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:49.401701 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 19:30:49.401938 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:30:49.402881 22039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:49.491053 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21838
I20260810 19:30:49.519006 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 21229
I20260810 19:30:49.520170 21335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:49.629706 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 21229
2026-08-10T19:30:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4630 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T19:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:49Z Disabled control of system clock
I20260810 19:30:49.722934 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44173
--webserver_interface=127.19.68.62
--webserver_port=0
--builtin_ntp_servers=127.19.68.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:50.053247 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:50.053539 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:50.053653 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:50.061801 22128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:50.062000 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:50.062090 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:50.062187 22128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:50.062261 22128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:50.071527 22128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:44173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:50.072808 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:50.074661 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:50.085994 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:50.085961 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:50.086473 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:50.086952 22128 server_base.cc:1034] running on GCE node
I20260810 19:30:50.087651 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:50.088914 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:50.090098 22128 hybrid_clock.cc:648] HybridClock initialized: now 1786390250090041 us; error 50 us; skew 500 ppm
I20260810 19:30:50.090435 22128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:50.092573 22128 webserver.cc:460] Webserver started at http://127.19.68.62:40531/ using document root <none> and password file <none>
I20260810 19:30:50.093250 22128 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:50.093369 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:50.093675 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:50.096499 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "897aa46aa272449fb268382863f493aa"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "14020fdc4c7a8c9f50b76bdf60f2d08f"
server_key_iv: "1549b4ccbf9abb133bb09ab07345a95a"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.097355 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "897aa46aa272449fb268382863f493aa"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "14020fdc4c7a8c9f50b76bdf60f2d08f"
server_key_iv: "1549b4ccbf9abb133bb09ab07345a95a"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.102308 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:30:50.105666 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:50.106954 22128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:30:50.107113 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "897aa46aa272449fb268382863f493aa"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "14020fdc4c7a8c9f50b76bdf60f2d08f"
server_key_iv: "1549b4ccbf9abb133bb09ab07345a95a"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.107311 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:50.119024 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:50.119781 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:50.120040 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:50.137328 22128 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:44173
I20260810 19:30:50.137339 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:44173 every 8 connection(s)
I20260810 19:30:50.138624 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:50.142200 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22128
I20260810 19:30:50.142392 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:50.142418 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:50.142979 19728 external_mini_cluster.cc:1428] Setting key 3e2825f66650a6b57a9d41f54ad8faa5
I20260810 19:30:50.155874 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: Bootstrap starting.
I20260810 19:30:50.156271 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:50.144880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.19.68.62:44173)
0810 19:30:50.145881 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:50.145891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:50.145933 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:30:50.147431 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:50.147451 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:50.147895 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:50.148208 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:50.148797 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.150136 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.151961 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.152746 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.152911 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:50.154089 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:50.154097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:50.154110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:50.154613 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:50.154686 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:50.154707 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:50.154791 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:50.155016 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 19:30:50.155305 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":386,"threads_started":1}
I20260810 19:30:50.159010 22243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:50.159967 22243 log.cc:824] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:50.162457 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: No bootstrap required, opened a new log
I20260810 19:30:50.169966 22243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } }
I20260810 19:30:50.170297 22243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:50.170390 22243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897aa46aa272449fb268382863f493aa, State: Initialized, Role: FOLLOWER
I20260810 19:30:50.170912 22243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } }
I20260810 19:30:50.171135 22243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:50.171262 22243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:50.171401 22243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:50.173676 22243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } }
I20260810 19:30:50.174167 22243 leader_election.cc:302] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897aa46aa272449fb268382863f493aa; no voters:
I20260810 19:30:50.174705 22243 leader_election.cc:288] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:50.174845 22254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:50.175575 22254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [term 1 LEADER]: Becoming Leader. State: Replica: 897aa46aa272449fb268382863f493aa, State: Running, Role: LEADER
I20260810 19:30:50.176035 22254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } }
I20260810 19:30:50.176604 22243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:50.178912 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897aa46aa272449fb268382863f493aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } } }
I20260810 19:30:50.179425 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:50.178939 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897aa46aa272449fb268382863f493aa. Latest consensus state: current_term: 1 leader_uuid: "897aa46aa272449fb268382863f493aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897aa46aa272449fb268382863f493aa" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 44173 } } }
I20260810 19:30:50.179898 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:50.181777 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:50.186830 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:50.195492 22262 catalog_manager.cc:1347] Generated new cluster ID: 50a421844e704a48ab05bd842628b759
I20260810 19:30:50.195593 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:50.207226 22262 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:50.208479 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:50.214491 22262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: Generated new TSK 0
I20260810 19:30:50.215502 22262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:50.229533 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.1:0
--local_ip_for_outbound_sockets=127.19.68.1
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--builtin_ntp_servers=127.19.68.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:30:50.560865 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:50.561187 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:50.561329 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:50.568418 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:50.568503 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:50.568650 22287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.1
I20260810 19:30:50.577491 22287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.68.1
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.1
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:50.578769 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:50.580477 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:50.592916 22304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:50.593571 22296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:50.593730 22307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:50.593895 22287 server_base.cc:1034] running on GCE node
I20260810 19:30:50.594498 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:50.595373 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:50.596565 22287 hybrid_clock.cc:648] HybridClock initialized: now 1786390250596508 us; error 51 us; skew 500 ppm
I20260810 19:30:50.596843 22287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:50.598956 22287 webserver.cc:460] Webserver started at http://127.19.68.1:39257/ using document root <none> and password file <none>
I20260810 19:30:50.599589 22287 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:50.599684 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:50.599988 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:50.602777 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data/instance:
uuid: "915f9584b379413492bb6d96842165b6"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "baa2d6674bf1a0d9f1ad29a1a69818c4"
server_key_iv: "50a018b6f9a322d71178b90f5938a85b"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.603499 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance:
uuid: "915f9584b379413492bb6d96842165b6"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "baa2d6674bf1a0d9f1ad29a1a69818c4"
server_key_iv: "50a018b6f9a322d71178b90f5938a85b"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.608238 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 19:30:50.611610 22318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:50.612826 22287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:30:50.612972 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal
uuid: "915f9584b379413492bb6d96842165b6"
format_stamp: "Formatted at 2026-08-10 19:30:50 on dist-test-slave-2j7r"
server_key: "baa2d6674bf1a0d9f1ad29a1a69818c4"
server_key_iv: "50a018b6f9a322d71178b90f5938a85b"
server_key_version: "encryptionkey@0"
I20260810 19:30:50.613220 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:50.642282 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:50.643141 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:50.643424 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:50.644412 22287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:50.645929 22287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:50.646032 22287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:50.646138 22287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:50.646214 22287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:50.675850 22287 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.1:43171
I20260810 19:30:50.675894 22514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.1:43171 every 8 connection(s)
I20260810 19:30:50.677279 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/data/info.pb
I20260810 19:30:50.681516 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22287
I20260810 19:30:50.681684 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-0/wal/instance
I20260810 19:30:50.682125 19728 external_mini_cluster.cc:1428] Setting key 9088fc4d61db8af3db87038b8cb232ee
I20260810 19:30:50.684885 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.2:0
--local_ip_for_outbound_sockets=127.19.68.2
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--builtin_ntp_servers=127.19.68.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:50.690056 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:50.679753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.1:57315 (local address 127.19.68.62:44173)
0810 19:30:50.680221 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:50.680231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:50.681451 (+ 1220us) server_negotiation.cc:407] Connection header received
0810 19:30:50.683617 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:50.683622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:50.683709 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:50.683813 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:50.684935 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.685886 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.687380 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.687952 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.688070 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:50.689315 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:50.689321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:50.689325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:50.689345 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:50.689404 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:50.689419 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:50.689512 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:50.689668 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 19:30:50.689759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":212,"threads_started":1}
I20260810 19:30:50.690907 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:50.680358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44173 (local address 127.19.68.1:57315)
0810 19:30:50.681301 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:50.681339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:50.683338 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:50.684096 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:50.684108 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:50.684209 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:50.684745 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:50.684761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.686038 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.686042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:50.687239 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:50.687252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:50.688148 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:50.688153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:50.688353 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:50.688973 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:50.688976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:50.689014 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:50.689122 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:50.689507 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:50.689513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:50.689517 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:30:50.689641 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 19:30:50.689930 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260810 19:30:50.692256 22516 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44173
I20260810 19:30:50.692566 22516 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:50.693410 22516 heartbeater.cc:511] Master 127.19.68.62:44173 requested a full tablet report, sending...
I20260810 19:30:50.695411 22182 ts_manager.cc:194] Registered new tserver with Master: 915f9584b379413492bb6d96842165b6 (127.19.68.1:43171)
I20260810 19:30:50.696828 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.1:57315
W20260810 19:30:51.016215 22521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:51.016467 22521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:51.016549 22521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:51.023613 22521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:51.023706 22521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:51.023823 22521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.2
I20260810 19:30:51.032473 22521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.68.2
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.2
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:51.033660 22521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:51.035395 22521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:51.047740 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:51.047706 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:51.047803 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:51.048728 22521 server_base.cc:1034] running on GCE node
I20260810 19:30:51.049336 22521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:51.050170 22521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:51.051379 22521 hybrid_clock.cc:648] HybridClock initialized: now 1786390251051346 us; error 25 us; skew 500 ppm
I20260810 19:30:51.051654 22521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:51.053732 22521 webserver.cc:460] Webserver started at http://127.19.68.2:46765/ using document root <none> and password file <none>
I20260810 19:30:51.054374 22521 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:51.054466 22521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:51.054721 22521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:51.057526 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data/instance:
uuid: "f779c8c59b0d492da4449637cf34aa37"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "ebe97dc98c736591d6b709c7894e93b4"
server_key_iv: "495aac2a9cca8869faf5b2b51fc72f99"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.058264 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance:
uuid: "f779c8c59b0d492da4449637cf34aa37"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "ebe97dc98c736591d6b709c7894e93b4"
server_key_iv: "495aac2a9cca8869faf5b2b51fc72f99"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.063201 22521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 19:30:51.066637 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.067907 22521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:30:51.068044 22521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal
uuid: "f779c8c59b0d492da4449637cf34aa37"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "ebe97dc98c736591d6b709c7894e93b4"
server_key_iv: "495aac2a9cca8869faf5b2b51fc72f99"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.068216 22521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:51.090610 22521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:51.091387 22521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:51.091622 22521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:51.092574 22521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:51.094084 22521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:51.094172 22521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.094250 22521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:51.094306 22521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.124539 22521 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.2:37951
I20260810 19:30:51.124667 22732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.2:37951 every 8 connection(s)
I20260810 19:30:51.126003 22521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/data/info.pb
I20260810 19:30:51.127072 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22521
I20260810 19:30:51.127256 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal/instance
I20260810 19:30:51.127825 19728 external_mini_cluster.cc:1428] Setting key c1c357e3a6594fbbfc9d23eda364b99e
I20260810 19:30:51.130923 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.68.3:0
--local_ip_for_outbound_sockets=127.19.68.3
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--builtin_ntp_servers=127.19.68.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:30:51.139209 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.128776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:39555 (local address 127.19.68.62:44173)
0810 19:30:51.129006 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.129013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.130577 (+ 1564us) server_negotiation.cc:407] Connection header received
0810 19:30:51.132413 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.132419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.132505 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.132594 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:51.133733 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.134476 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.136027 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.136418 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.136548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.138352 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:51.138358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:51.138363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:51.138392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:51.138432 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:51.138448 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:51.138544 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.138787 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.138901 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:30:51.140539 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.129422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44173 (local address 127.19.68.2:39555)
0810 19:30:51.130420 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.130468 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.132147 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.132831 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.132839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.132945 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:51.133564 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.133582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.134636 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.134640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.135873 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.135886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.136836 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.136844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.137165 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.137950 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:51.137954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:51.138010 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:51.138121 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:51.138556 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:51.138567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:51.138574 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.138754 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.139154 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":345,"threads_started":1}
I20260810 19:30:51.142047 22733 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44173
I20260810 19:30:51.142373 22733 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:51.143217 22733 heartbeater.cc:511] Master 127.19.68.62:44173 requested a full tablet report, sending...
I20260810 19:30:51.144788 22182 ts_manager.cc:194] Registered new tserver with Master: f779c8c59b0d492da4449637cf34aa37 (127.19.68.2:37951)
I20260810 19:30:51.145880 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.2:39555
W20260810 19:30:51.466148 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:51.466439 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:51.466574 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:51.473920 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:51.474026 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:51.474169 22736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.68.3
I20260810 19:30:51.483736 22736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.68.3
--webserver_port=0
--tserver_master_addrs=127.19.68.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.68.3
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:51.485296 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:51.487100 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:51.499643 22749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:51.499682 22747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:51.500025 22746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:51.500377 22736 server_base.cc:1034] running on GCE node
I20260810 19:30:51.500979 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:51.501935 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:51.503150 22736 hybrid_clock.cc:648] HybridClock initialized: now 1786390251503106 us; error 31 us; skew 500 ppm
I20260810 19:30:51.503465 22736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:30:51.505671 22736 webserver.cc:460] Webserver started at http://127.19.68.3:33027/ using document root <none> and password file <none>
I20260810 19:30:51.506479 22736 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:51.506600 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:51.506965 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:51.510380 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data/instance:
uuid: "f258e7403d184d14999b4fe204a4b7a1"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "4965af9cc6e9b8175361bde06c589bea"
server_key_iv: "e1038a7bbea16bb822c869d641ebba75"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.511157 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance:
uuid: "f258e7403d184d14999b4fe204a4b7a1"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "4965af9cc6e9b8175361bde06c589bea"
server_key_iv: "e1038a7bbea16bb822c869d641ebba75"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.516340 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:30:51.519835 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.521162 22736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:30:51.521301 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal
uuid: "f258e7403d184d14999b4fe204a4b7a1"
format_stamp: "Formatted at 2026-08-10 19:30:51 on dist-test-slave-2j7r"
server_key: "4965af9cc6e9b8175361bde06c589bea"
server_key_iv: "e1038a7bbea16bb822c869d641ebba75"
server_key_version: "encryptionkey@0"
I20260810 19:30:51.521492 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:51.539614 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:51.540362 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:51.540598 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:51.541587 22736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:30:51.542966 22736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:30:51.543056 22736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.543134 22736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:30:51.543188 22736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:51.573369 22736 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.3:46649
I20260810 19:30:51.573395 22976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.3:46649 every 8 connection(s)
I20260810 19:30:51.574633 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/data/info.pb
I20260810 19:30:51.575723 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 22736
I20260810 19:30:51.575906 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal/instance
I20260810 19:30:51.576454 19728 external_mini_cluster.cc:1428] Setting key 634f85b6ecc3923d794b97ca4672b1c0
I20260810 19:30:51.586512 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.577297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.3:39475 (local address 127.19.68.62:44173)
0810 19:30:51.577488 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.577494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.579011 (+ 1517us) server_negotiation.cc:407] Connection header received
0810 19:30:51.580796 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.580800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.580866 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.580938 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:51.581884 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.582644 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.584192 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.584611 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.584703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.585877 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:51.585882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:51.585885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:51.585902 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:51.585936 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:51.585959 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:51.586022 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.586181 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.586258 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:30:51.587366 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.577810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:44173 (local address 127.19.68.3:39475)
0810 19:30:51.578832 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.578899 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.580558 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.581119 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.581130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.581212 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:30:51.581750 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.581765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.582780 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.582783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.584068 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.584080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.584771 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.584775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.584986 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.585596 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:30:51.585598 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:30:51.585635 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:30:51.585708 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:30:51.586026 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:30:51.586033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:30:51.586038 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.586158 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.586424 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":308,"threads_started":1}
I20260810 19:30:51.588483 22979 heartbeater.cc:347] Connected to a master server at 127.19.68.62:44173
I20260810 19:30:51.588723 22979 heartbeater.cc:464] Registering TS with master...
I20260810 19:30:51.589442 22979 heartbeater.cc:511] Master 127.19.68.62:44173 requested a full tablet report, sending...
I20260810 19:30:51.590803 22182 ts_manager.cc:194] Registered new tserver with Master: f258e7403d184d14999b4fe204a4b7a1 (127.19.68.3:46649)
I20260810 19:30:51.591675 19728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:30:51.591840 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.68.3:39475
I20260810 19:30:51.600672 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.595545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.19.68.62:44173)
0810 19:30:51.595691 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.595695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.596141 (+ 446us) server_negotiation.cc:407] Connection header received
0810 19:30:51.596264 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.596267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.596341 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.596416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:51.596856 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.597544 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.598864 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.599232 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.599313 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.600152 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:51.600156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:51.600159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:51.600176 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:51.600206 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:51.600217 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:51.600279 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.600381 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.600453 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:30:51.604041 22182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:30:51.611088 22182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:30:51.638221 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.622391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:46649 (local address 127.0.0.1:36534)
0810 19:30:51.628205 (+ 5814us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.628249 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.628456 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.629603 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.629612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.629646 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:51.630072 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.630085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.632191 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.632197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.634764 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.634782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.637207 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.637215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.637458 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.637512 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.637645 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.637849 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5540,"thread_start_us":5376,"threads_started":1}
I20260810 19:30:51.639770 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.622640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.19.68.3:46649)
0810 19:30:51.623434 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.623443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.628657 (+ 5214us) server_negotiation.cc:407] Connection header received
0810 19:30:51.628770 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.628778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.629037 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.629261 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:51.630316 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.632015 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.635017 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.636581 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.636718 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.637037 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.638836 (+ 1799us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.639040 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":428,"threads_started":1}
I20260810 19:30:51.644048 22849 tablet_service.cc:1467] Processing CreateTablet for tablet c73015ecfe4648469ac6c7638698f78c (DEFAULT_TABLE table=test-table [id=2ab8cf4e03164bedb2d51b3e46297eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:51.645726 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73015ecfe4648469ac6c7638698f78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:51.649030 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.622792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.2:37951 (local address 127.0.0.1:42130)
0810 19:30:51.638535 (+ 15743us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.638557 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.638667 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.642749 (+ 4082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.642754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.642784 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:51.642965 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.642976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.645467 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.645473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.647044 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.647059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.648266 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.648272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.648460 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.648493 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.648574 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.648721 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15612,"thread_start_us":22391,"threads_started":1}
I20260810 19:30:51.649121 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.623036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.19.68.2:37951)
0810 19:30:51.641464 (+ 18428us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.641474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.641511 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:30:51.642001 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.642016 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.642291 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.643150 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:51.643422 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.644940 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.647194 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.648098 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.648216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.648461 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.648638 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.648798 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18219,"thread_start_us":18072,"threads_started":1}
I20260810 19:30:51.651600 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.628390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60760)
0810 19:30:51.638586 (+ 10196us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.638605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.638720 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.644363 (+ 5643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.644368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.644393 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:51.644872 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.644886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.646710 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.646715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.648761 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.648777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.650821 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.650828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.651033 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.651065 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.651155 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.651277 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10060,"mutex_wait_us":1,"thread_start_us":10144,"threads_started":1}
I20260810 19:30:51.651789 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.628720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.19.68.1:43171)
0810 19:30:51.642400 (+ 13680us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.642409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.642436 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:30:51.642525 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.642532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.643913 (+ 1381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.644098 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:51.645062 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.646547 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.648991 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.650339 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.650476 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.650686 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.651295 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.651460 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13517,"thread_start_us":14903,"threads_started":1}
I20260810 19:30:51.652678 22617 tablet_service.cc:1467] Processing CreateTablet for tablet c73015ecfe4648469ac6c7638698f78c (DEFAULT_TABLE table=test-table [id=2ab8cf4e03164bedb2d51b3e46297eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:51.654068 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73015ecfe4648469ac6c7638698f78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:51.655467 22395 tablet_service.cc:1467] Processing CreateTablet for tablet c73015ecfe4648469ac6c7638698f78c (DEFAULT_TABLE table=test-table [id=2ab8cf4e03164bedb2d51b3e46297eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:30:51.656750 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73015ecfe4648469ac6c7638698f78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:51.658711 23009 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Bootstrap starting.
I20260810 19:30:51.662271 23009 tablet_bootstrap.cc:654] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:51.663201 23009 log.cc:824] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:51.665644 23009 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: No bootstrap required, opened a new log
I20260810 19:30:51.665894 23009 ts_tablet_manager.cc:1397] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 19:30:51.668669 23011 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Bootstrap starting.
I20260810 19:30:51.670686 23014 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: Bootstrap starting.
I20260810 19:30:51.672593 23011 tablet_bootstrap.cc:654] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:51.673645 23014 tablet_bootstrap.cc:654] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:51.673918 23011 log.cc:824] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:51.674584 23014 log.cc:824] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:51.675879 23009 raft_consensus.cc:348] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.676365 23009 raft_consensus.cc:374] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:51.676481 23009 raft_consensus.cc:729] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f258e7403d184d14999b4fe204a4b7a1, State: Initialized, Role: FOLLOWER
I20260810 19:30:51.677299 23009 consensus_queue.cc:260] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.678534 22979 heartbeater.cc:503] Master 127.19.68.62:44173 was elected leader, sending a full tablet report...
I20260810 19:30:51.679111 23009 ts_tablet_manager.cc:1428] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 19:30:51.681113 23011 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: No bootstrap required, opened a new log
I20260810 19:30:51.681378 23011 ts_tablet_manager.cc:1397] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260810 19:30:51.687299 23014 tablet_bootstrap.cc:492] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: No bootstrap required, opened a new log
I20260810 19:30:51.687628 23014 ts_tablet_manager.cc:1397] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260810 19:30:51.692088 23011 raft_consensus.cc:348] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.692402 23011 raft_consensus.cc:374] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:51.692493 23011 raft_consensus.cc:729] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f779c8c59b0d492da4449637cf34aa37, State: Initialized, Role: FOLLOWER
I20260810 19:30:51.692999 23011 consensus_queue.cc:260] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.694048 22733 heartbeater.cc:503] Master 127.19.68.62:44173 was elected leader, sending a full tablet report...
I20260810 19:30:51.694814 23011 ts_tablet_manager.cc:1428] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260810 19:30:51.695322 23014 raft_consensus.cc:348] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.695712 23014 raft_consensus.cc:374] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:51.695820 23014 raft_consensus.cc:729] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915f9584b379413492bb6d96842165b6, State: Initialized, Role: FOLLOWER
I20260810 19:30:51.696302 23014 consensus_queue.cc:260] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.697310 22516 heartbeater.cc:503] Master 127.19.68.62:44173 was elected leader, sending a full tablet report...
I20260810 19:30:51.697847 23014 ts_tablet_manager.cc:1428] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260810 19:30:51.709300 23020 raft_consensus.cc:482] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:30:51.709545 23020 raft_consensus.cc:504] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.711063 23020 leader_election.cc:288] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f258e7403d184d14999b4fe204a4b7a1 (127.19.68.3:46649), 915f9584b379413492bb6d96842165b6 (127.19.68.1:43171)
I20260810 19:30:51.717954 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.711482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:49249 (local address 127.19.68.3:46649)
0810 19:30:51.711715 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.711721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.712109 (+ 388us) server_negotiation.cc:407] Connection header received
0810 19:30:51.712320 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.712326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.712410 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.712509 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:51.713145 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.714070 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.716033 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.717023 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.717162 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.717260 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.717600 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.717727 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:30:51.718097 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.712276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.68.2:48831 (local address 127.19.68.1:43171)
0810 19:30:51.712462 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.712468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.712796 (+ 328us) server_negotiation.cc:407] Connection header received
0810 19:30:51.712919 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.712924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.713006 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.713110 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:30:51.713656 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.714386 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.716259 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.717206 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.717322 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.717405 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.717741 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.717862 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:30:51.718114 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.711400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.3:46649 (local address 127.19.68.2:49249)
0810 19:30:51.712006 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.712022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.712168 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.712561 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.712566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.712595 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:51.712975 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.712989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.714194 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.714199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.715859 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.715875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.717232 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.717238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.717410 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.717473 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.717569 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.717701 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":48,"thread_start_us":300,"threads_started":1}
I20260810 19:30:51.718287 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.712150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.19.68.2:48831)
0810 19:30:51.712699 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.712715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:30:51.712815 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:30:51.713147 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:30:51.713151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:30:51.713175 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:30:51.713522 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.713534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.714516 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.714519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.716088 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:30:51.716101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.717406 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.717412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:30:51.717596 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.717627 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:30:51.717706 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:30:51.717822 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":25,"thread_start_us":244,"threads_started":1}
I20260810 19:30:51.719154 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73015ecfe4648469ac6c7638698f78c" candidate_uuid: "f779c8c59b0d492da4449637cf34aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f258e7403d184d14999b4fe204a4b7a1" is_pre_election: true
I20260810 19:30:51.719424 22437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73015ecfe4648469ac6c7638698f78c" candidate_uuid: "f779c8c59b0d492da4449637cf34aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915f9584b379413492bb6d96842165b6" is_pre_election: true
I20260810 19:30:51.719753 22889 raft_consensus.cc:2454] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f779c8c59b0d492da4449637cf34aa37 in term 0.
I20260810 19:30:51.719959 22437 raft_consensus.cc:2454] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f779c8c59b0d492da4449637cf34aa37 in term 0.
I20260810 19:30:51.720413 22550 leader_election.cc:302] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f258e7403d184d14999b4fe204a4b7a1, f779c8c59b0d492da4449637cf34aa37; no voters:
I20260810 19:30:51.720714 23020 raft_consensus.cc:2781] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:30:51.720844 23020 raft_consensus.cc:482] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:30:51.720964 23020 raft_consensus.cc:3037] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:51.723119 23020 raft_consensus.cc:504] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.723627 23020 leader_election.cc:288] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [CANDIDATE]: Term 1 election: Requested vote from peers f258e7403d184d14999b4fe204a4b7a1 (127.19.68.3:46649), 915f9584b379413492bb6d96842165b6 (127.19.68.1:43171)
I20260810 19:30:51.724041 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73015ecfe4648469ac6c7638698f78c" candidate_uuid: "f779c8c59b0d492da4449637cf34aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f258e7403d184d14999b4fe204a4b7a1"
I20260810 19:30:51.724066 22437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73015ecfe4648469ac6c7638698f78c" candidate_uuid: "f779c8c59b0d492da4449637cf34aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915f9584b379413492bb6d96842165b6"
I20260810 19:30:51.724220 22437 raft_consensus.cc:3037] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:51.724215 22889 raft_consensus.cc:3037] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:51.726471 22437 raft_consensus.cc:2454] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f779c8c59b0d492da4449637cf34aa37 in term 1.
I20260810 19:30:51.726677 22889 raft_consensus.cc:2454] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f779c8c59b0d492da4449637cf34aa37 in term 1.
I20260810 19:30:51.726823 22550 leader_election.cc:302] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915f9584b379413492bb6d96842165b6, f779c8c59b0d492da4449637cf34aa37; no voters:
I20260810 19:30:51.727114 23020 raft_consensus.cc:2781] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:51.727520 23020 raft_consensus.cc:686] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 1 LEADER]: Becoming Leader. State: Replica: f779c8c59b0d492da4449637cf34aa37, State: Running, Role: LEADER
I20260810 19:30:51.727996 23020 consensus_queue.cc:237] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } }
I20260810 19:30:51.732204 22181 catalog_manager.cc:5495] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 reported cstate change: term changed from 0 to 1, leader changed from <none> to f779c8c59b0d492da4449637cf34aa37 (127.19.68.2). New cstate: current_term: 1 leader_uuid: "f779c8c59b0d492da4449637cf34aa37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f779c8c59b0d492da4449637cf34aa37" member_type: VOTER last_known_addr { host: "127.19.68.2" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 } health_report { overall_health: UNKNOWN } } }
I20260810 19:30:51.746758 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.741559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.19.68.2:37951)
0810 19:30:51.741748 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.741754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.741857 (+ 103us) server_negotiation.cc:407] Connection header received
0810 19:30:51.742087 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.742091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.742167 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.742289 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:30:51.742644 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.743329 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.744705 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.745167 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.745278 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.745734 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:30:51.746200 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:30:51.746329 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.746450 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.746525 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:30:51.754216 22437 raft_consensus.cc:1261] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 1 FOLLOWER]: Refusing update from remote peer f779c8c59b0d492da4449637cf34aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:51.754262 22889 raft_consensus.cc:1261] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 1 FOLLOWER]: Refusing update from remote peer f779c8c59b0d492da4449637cf34aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:30:51.755091 23029 consensus_queue.cc:1035] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f258e7403d184d14999b4fe204a4b7a1" member_type: VOTER last_known_addr { host: "127.19.68.3" port: 46649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:51.755407 23020 consensus_queue.cc:1035] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "915f9584b379413492bb6d96842165b6" member_type: VOTER last_known_addr { host: "127.19.68.1" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:30:51.778213 22182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:51.778481 22182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:30:51.781793 22182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897aa46aa272449fb268382863f493aa: Sending DeleteTablet for 3 replicas of tablet c73015ecfe4648469ac6c7638698f78c
I20260810 19:30:51.782994 22617 tablet_service.cc:1514] Processing DeleteTablet for tablet c73015ecfe4648469ac6c7638698f78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:51 UTC) from {username='slave'} at 127.0.0.1:42130
I20260810 19:30:51.782987 22849 tablet_service.cc:1514] Processing DeleteTablet for tablet c73015ecfe4648469ac6c7638698f78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:51 UTC) from {username='slave'} at 127.0.0.1:36534
I20260810 19:30:51.783071 22395 tablet_service.cc:1514] Processing DeleteTablet for tablet c73015ecfe4648469ac6c7638698f78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:30:51 UTC) from {username='slave'} at 127.0.0.1:60760
I20260810 19:30:51.784940 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22287
I20260810 19:30:51.795785 23045 tablet_replica.cc:330] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: stopping tablet replica
I20260810 19:30:51.796360 23047 tablet_replica.cc:330] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: stopping tablet replica
I20260810 19:30:51.796950 23047 raft_consensus.cc:2229] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:51.797502 23045 raft_consensus.cc:2229] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:30:51.798206 23045 raft_consensus.cc:2258] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:51.798771 23047 raft_consensus.cc:2258] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:51.800987 23047 ts_tablet_manager.cc:1905] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:51.802332 23045 ts_tablet_manager.cc:1905] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:30:51.802956 23048 tablet_replica.cc:330] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6: stopping tablet replica
I20260810 19:30:51.809641 23047 ts_tablet_manager.cc:1918] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:51.809859 23047 log.cc:1196] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-2/wal/wals/c73015ecfe4648469ac6c7638698f78c
I20260810 19:30:51.810247 23048 raft_consensus.cc:2229] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:30:51.810420 23047 ts_tablet_manager.cc:1939] T c73015ecfe4648469ac6c7638698f78c P f258e7403d184d14999b4fe204a4b7a1: Deleting consensus metadata
I20260810 19:30:51.811208 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.788017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.19.68.1:43171)
0810 19:30:51.788197 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.788203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.802827 (+ 14624us) server_negotiation.cc:407] Connection header received
0810 19:30:51.803318 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.803323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.803436 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.803512 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:51.804179 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.805315 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.807094 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.807750 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.807846 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.809294 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:51.809307 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:51.809324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:51.810133 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:51.810194 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:51.810239 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:51.810341 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.810664 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.810786 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:30:51.812064 22504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:51.810992 23048 raft_consensus.cc:2258] T c73015ecfe4648469ac6c7638698f78c P 915f9584b379413492bb6d96842165b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:30:51.812577 22150 catalog_manager.cc:4841] TS f258e7403d184d14999b4fe204a4b7a1 (127.19.68.3:46649): tablet c73015ecfe4648469ac6c7638698f78c (table test-table [id=2ab8cf4e03164bedb2d51b3e46297eb0]) successfully deleted
I20260810 19:30:51.812819 23045 ts_tablet_manager.cc:1918] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:30:51.812968 23045 log.cc:1196] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Deleting WAL directory at /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389929706493-19728-0/minicluster-data/ts-1/wal/wals/c73015ecfe4648469ac6c7638698f78c
I20260810 19:30:51.813709 23045 ts_tablet_manager.cc:1939] T c73015ecfe4648469ac6c7638698f78c P f779c8c59b0d492da4449637cf34aa37: Deleting consensus metadata
I20260810 19:30:51.816296 22158 catalog_manager.cc:4841] TS f779c8c59b0d492da4449637cf34aa37 (127.19.68.2:37951): tablet c73015ecfe4648469ac6c7638698f78c (table test-table [id=2ab8cf4e03164bedb2d51b3e46297eb0]) successfully deleted
I20260810 19:30:51.900800 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22287
I20260810 19:30:51.925202 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22521
W20260810 19:30:51.925267 22150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.68.1:43171 (error 108)
I20260810 19:30:51.927883 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.927421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60788)
0810 19:30:51.927609 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.927728 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 19:30:51.928198 22150 catalog_manager.cc:4584] TS 915f9584b379413492bb6d96842165b6 (127.19.68.1:43171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c73015ecfe4648469ac6c7638698f78c: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
I20260810 19:30:51.933799 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.926176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.19.68.2:37951)
0810 19:30:51.926387 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:51.926394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:51.926649 (+ 255us) server_negotiation.cc:407] Connection header received
0810 19:30:51.926734 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:51.926740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:51.926891 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:51.927005 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:51.927509 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.928749 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.930253 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:51.930967 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:51.931109 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:51.932080 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:51.932092 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:51.932109 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:51.932930 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:51.932994 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:51.933020 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:51.933157 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:51.933284 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 19:30:51.933409 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:30:51.934568 22725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:51.981834 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.981250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60800)
0810 19:30:51.981483 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.981611 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:30:51.982955 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:51.982536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60812)
0810 19:30:51.982704 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:51.982799 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:30:52.043629 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:52.043016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60828)
0810 19:30:52.043304 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:52.043419 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 19:30:52.044665 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:52.044278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60830)
0810 19:30:52.044440 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:52.044524 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:30:52.047026 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22521
I20260810 19:30:52.071611 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22736
I20260810 19:30:52.078320 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:52.072437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.19.68.3:46649)
0810 19:30:52.072616 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:52.072620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:52.072677 (+ 57us) server_negotiation.cc:407] Connection header received
0810 19:30:52.072834 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:52.072838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:52.072935 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:52.073025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:52.073417 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:52.074154 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:52.075404 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:52.075783 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:52.075872 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:52.076861 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:52.076870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:52.076885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:52.077671 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:52.077721 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:52.077741 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:52.077822 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:52.077963 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 19:30:52.078056 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:30:52.078951 22970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:52.125367 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:52.124763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60838)
0810 19:30:52.125004 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:52.125144 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 19:30:52.126559 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:52.126103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.1:43171 (local address 127.0.0.1:60854)
0810 19:30:52.126332 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 19:30:52.126418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 19:30:52.167052 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22736
I20260810 19:30:52.188556 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 22128
I20260810 19:30:52.189635 22234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:52.296998 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 22128
2026-08-10T19:30:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2666 ms)
[----------] 6 tests from EncryptionPolicyTest (22314 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:52 dist-test-slave-2j7r krb5kdc[23064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:52 dist-test-slave-2j7r krb5kdc[23064](info): set up 2 sockets
Aug 10 19:30:52 dist-test-slave-2j7r krb5kdc[23064](info): commencing operation
krb5kdc: starting...
W20260810 19:30:53.579643 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.006s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:53 dist-test-slave-2j7r krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:30:53.886896 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:54.220319 23088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:54.220633 23088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:54.220772 23088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:54.228071 23088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:54.228204 23088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:54.228291 23088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:54.228374 23088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:54.228432 23088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:54.238449 23088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:54.239869 23088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:54.241201 23088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:54.252230 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:54.252261 23108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:54.252333 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:54.253422 23088 server_base.cc:1034] running on GCE node
I20260810 19:30:54.254118 23088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:54.255430 23088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:54.256639 23088 hybrid_clock.cc:648] HybridClock initialized: now 1786390254256599 us; error 26 us; skew 500 ppm
I20260810 19:30:54.256984 23088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:54 dist-test-slave-2j7r krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:54.265044 23088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:30:54.266850 23088 webserver.cc:460] Webserver started at http://127.0.0.1:34929/ using document root <none> and password file <none>
I20260810 19:30:54.267584 23088 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:54.267725 23088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:54.268066 23088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:54.271363 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "c9d70ce971084be794d091cb88308862"
format_stamp: "Formatted at 2026-08-10 19:30:54 on dist-test-slave-2j7r"
server_key: "8ec031c4c81668c11d32d6ebf7790dd4"
server_key_iv: "f4a0c34f408377fdde25e0b10c9f378b"
server_key_version: "encryptionkey@0"
I20260810 19:30:54.272215 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "c9d70ce971084be794d091cb88308862"
format_stamp: "Formatted at 2026-08-10 19:30:54 on dist-test-slave-2j7r"
server_key: "8ec031c4c81668c11d32d6ebf7790dd4"
server_key_iv: "f4a0c34f408377fdde25e0b10c9f378b"
server_key_version: "encryptionkey@0"
I20260810 19:30:54.277827 23088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 19:30:54.281292 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:54.282611 23088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:30:54.282783 23088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "c9d70ce971084be794d091cb88308862"
format_stamp: "Formatted at 2026-08-10 19:30:54 on dist-test-slave-2j7r"
server_key: "8ec031c4c81668c11d32d6ebf7790dd4"
server_key_iv: "f4a0c34f408377fdde25e0b10c9f378b"
server_key_version: "encryptionkey@0"
I20260810 19:30:54.282964 23088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:54.306288 23088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:54.310190 23088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:54.310456 23088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:54.327759 23088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38949
I20260810 19:30:54.327791 23203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38949 every 8 connection(s)
I20260810 19:30:54.329056 23088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:54.332605 23204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:54.336462 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23088
I20260810 19:30:54.336676 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:54.337512 19728 external_mini_cluster.cc:1428] Setting key a4ea1beee23c42eb3718fcc1dd5327fe
I20260810 19:30:54.343930 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862: Bootstrap starting.
I20260810 19:30:54.347548 23204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:54.348658 23204 log.cc:824] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:54.351766 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862: No bootstrap required, opened a new log
Aug 10 19:30:54 dist-test-slave-2j7r krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:30:54.362051 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:54.339448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45507 (local address 127.0.0.1:38949)
0810 19:30:54.340134 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:54.340146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:54.340189 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:30:54.342503 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:54.342524 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:54.342880 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:54.343183 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:54.344895 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:54.346270 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:54.348063 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:54.348720 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:54.348882 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:54.353606 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:54.353617 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:54.353634 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:54.353682 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:54.357604 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:54.358376 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:54.358385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:54.358393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:54.358597 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:54.359020 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:54.359024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:54.359027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:54.359712 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:54.360061 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:54.360385 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 19:30:54.360724 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":299,"threads_started":1}
I20260810 19:30:54.361835 23204 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260810 19:30:54.362274 23204 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:54.362383 23204 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d70ce971084be794d091cb88308862, State: Initialized, Role: FOLLOWER
I20260810 19:30:54.362942 23204 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260810 19:30:54.363161 23204 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:54.363298 23204 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:54.363468 23204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:54.365940 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:30:54.365896 23204 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260810 19:30:54.366418 23204 leader_election.cc:302] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d70ce971084be794d091cb88308862; no voters:
I20260810 19:30:54.367046 23204 leader_election.cc:288] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:54.367133 23213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:54.368155 23213 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [term 1 LEADER]: Becoming Leader. State: Replica: c9d70ce971084be794d091cb88308862, State: Running, Role: LEADER
I20260810 19:30:54.368667 23213 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260810 19:30:54.369177 23204 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:54.371438 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d70ce971084be794d091cb88308862. Latest consensus state: current_term: 1 leader_uuid: "c9d70ce971084be794d091cb88308862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } } }
I20260810 19:30:54.371460 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d70ce971084be794d091cb88308862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70ce971084be794d091cb88308862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } } }
I20260810 19:30:54.371883 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:54.371883 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:54.373621 23223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:54.384332 23223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:54.412261 23223 catalog_manager.cc:1347] Generated new cluster ID: fcc296b436134ad3894bf5d185c4875d
I20260810 19:30:54.412395 23223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:54.413563 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:54.393411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38177 (local address 127.0.0.1:38949)
0810 19:30:54.393627 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:54.393633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:54.393660 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:30:54.393985 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:54.393991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:54.394079 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:54.394154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:54.396515 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:54.397793 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:54.399812 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:54.400575 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:54.400685 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:54.404511 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:54.404519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:54.404525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:30:54.404577 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:54.408446 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:54.409566 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:54.409572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:54.409577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:54.409729 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:30:54.410361 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:30:54.410367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:30:54.410371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:30:54.410835 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:54.411114 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:54.411841 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 19:30:54.412066 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:30:54.462288 23223 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:54.463600 23223 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:54.470824 23223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d70ce971084be794d091cb88308862: Generated new TSK 0
I20260810 19:30:54.471725 23223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:54.487977 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23088
I20260810 19:30:54.489275 23198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:30:54.586129 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23088
W20260810 19:30:54.613517 23060 connection.cc:535] client connection to 127.0.0.1:38949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38949): Connection reset by peer (error 104)
2026-08-10T19:30:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:54 dist-test-slave-2j7r krb5kdc[23252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:54 dist-test-slave-2j7r krb5kdc[23252](info): set up 2 sockets
Aug 10 19:30:54 dist-test-slave-2j7r krb5kdc[23252](info): commencing operation
krb5kdc: starting...
W20260810 19:30:55.857349 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:56 dist-test-slave-2j7r krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:30:56.172412 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:56.506505 23270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:56.506780 23270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:56.506919 23270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:56.514101 23270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:56.514214 23270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:56.514290 23270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:56.514379 23270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:56.514436 23270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:56.524837 23270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:56.526582 23270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:56.527911 23270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:56.539399 23277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:56.539434 23280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:56.539459 23278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:56.540030 23270 server_base.cc:1034] running on GCE node
I20260810 19:30:56.540791 23270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:56.542150 23270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:56.543325 23270 hybrid_clock.cc:648] HybridClock initialized: now 1786390256543274 us; error 35 us; skew 500 ppm
I20260810 19:30:56.543675 23270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:56 dist-test-slave-2j7r krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:56.546738 23270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:30:56.548169 23270 webserver.cc:460] Webserver started at http://127.0.0.1:46071/ using document root <none> and password file <none>
I20260810 19:30:56.548784 23270 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:56.548893 23270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:56.549187 23270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:56.552129 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "f55b1733a0a545f78cab7e90b06a2fc6"
format_stamp: "Formatted at 2026-08-10 19:30:56 on dist-test-slave-2j7r"
server_key: "f13fbeef8a0a5af153b621f9dc116afd"
server_key_iv: "b7e5d3df8178983bd444c39bbd963a44"
server_key_version: "encryptionkey@0"
I20260810 19:30:56.552862 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "f55b1733a0a545f78cab7e90b06a2fc6"
format_stamp: "Formatted at 2026-08-10 19:30:56 on dist-test-slave-2j7r"
server_key: "f13fbeef8a0a5af153b621f9dc116afd"
server_key_iv: "b7e5d3df8178983bd444c39bbd963a44"
server_key_version: "encryptionkey@0"
I20260810 19:30:56.557946 23270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:30:56.561352 23291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:56.562649 23270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:30:56.562801 23270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "f55b1733a0a545f78cab7e90b06a2fc6"
format_stamp: "Formatted at 2026-08-10 19:30:56 on dist-test-slave-2j7r"
server_key: "f13fbeef8a0a5af153b621f9dc116afd"
server_key_iv: "b7e5d3df8178983bd444c39bbd963a44"
server_key_version: "encryptionkey@0"
I20260810 19:30:56.562978 23270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:56.577486 23270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:56.581182 23270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:56.581434 23270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:56.598853 23270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44787
I20260810 19:30:56.598884 23376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44787 every 8 connection(s)
I20260810 19:30:56.600198 23270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:56.602532 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23270
I20260810 19:30:56.602689 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:56.603127 19728 external_mini_cluster.cc:1428] Setting key db1594c5a02070db799c0bd3f63b40d7
I20260810 19:30:56.604046 23379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:56.617061 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:56.604854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58525 (local address 127.0.0.1:44787)
0810 19:30:56.605691 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:56.605705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:56.605755 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:30:56.607499 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:56.607525 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:56.608022 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:56.608449 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:56.608942 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:56.610920 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:56.612178 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:56.613194 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:56.613363 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:56.614475 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:56.614486 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:56.614503 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:56.615219 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:56.615308 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:56.615341 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:56.615510 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:56.615657 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 19:30:56.615830 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":348,"threads_started":1}
I20260810 19:30:56.618464 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6: Bootstrap starting.
I20260810 19:30:56.620584 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:30:56.621650 23379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:56.622838 23379 log.cc:824] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:56.625986 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6: No bootstrap required, opened a new log
I20260810 19:30:56.632171 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:56.625577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60527 (local address 127.0.0.1:44787)
0810 19:30:56.625757 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:56.625764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:56.626221 (+ 457us) server_negotiation.cc:407] Connection header received
0810 19:30:56.626406 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:56.626409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:56.626470 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:56.626544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:56.627138 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:56.627928 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:56.629780 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:30:56.630231 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:30:56.630317 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:30:56.631447 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:56.631452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:56.631455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:56.631471 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:56.631520 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:56.631534 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:56.631644 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:56.631802 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 19:30:56.631848 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:30:56.635033 23379 raft_consensus.cc:348] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260810 19:30:56.635373 23379 raft_consensus.cc:374] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:56.635470 23379 raft_consensus.cc:729] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55b1733a0a545f78cab7e90b06a2fc6, State: Initialized, Role: FOLLOWER
I20260810 19:30:56.636003 23379 consensus_queue.cc:260] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260810 19:30:56.636207 23379 raft_consensus.cc:388] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:56.636328 23379 raft_consensus.cc:482] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:56.636472 23379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:56.638797 23379 raft_consensus.cc:504] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260810 19:30:56.639253 23379 leader_election.cc:302] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f55b1733a0a545f78cab7e90b06a2fc6; no voters:
I20260810 19:30:56.639794 23379 leader_election.cc:288] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:56.639922 23398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:56.640580 23398 raft_consensus.cc:686] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [term 1 LEADER]: Becoming Leader. State: Replica: f55b1733a0a545f78cab7e90b06a2fc6, State: Running, Role: LEADER
I20260810 19:30:56.641161 23398 consensus_queue.cc:237] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260810 19:30:56.641697 23379 sys_catalog.cc:564] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:56.644018 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f55b1733a0a545f78cab7e90b06a2fc6. Latest consensus state: current_term: 1 leader_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } } }
I20260810 19:30:56.644029 23399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55b1733a0a545f78cab7e90b06a2fc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } } }
I20260810 19:30:56.644451 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:56.644443 23399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:56.646447 23408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:56.651285 23408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:56.659351 23408 catalog_manager.cc:1347] Generated new cluster ID: 38d7692b272f4d4090f533f97b2f4cc7
I20260810 19:30:56.659469 23408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:56.671922 23408 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:56.673698 23408 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:56.695377 23408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f55b1733a0a545f78cab7e90b06a2fc6: Generated new TSK 0
I20260810 19:30:56.696434 23408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:56.705431 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23270
W20260810 19:30:56.706635 23371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58525
W20260810 19:30:56.707372 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:30:56.707548 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23270
W20260810 19:30:56.733999 23250 connection.cc:535] client connection to 127.0.0.1:44787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44787): Connection reset by peer (error 104)
2026-08-10T19:30:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2120 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:56 dist-test-slave-2j7r krb5kdc[23438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:56 dist-test-slave-2j7r krb5kdc[23438](info): set up 2 sockets
Aug 10 19:30:56 dist-test-slave-2j7r krb5kdc[23438](info): commencing operation
krb5kdc: starting...
W20260810 19:30:58.000797 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:30:58 dist-test-slave-2j7r krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:30:58.318007 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:30:58.663110 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:30:58.663421 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:30:58.663527 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:30:58.670881 23457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:30:58.670998 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:30:58.671057 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:30:58.671125 23457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:30:58.671175 23457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:30:58.680990 23457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:30:58.682476 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:30:58.683800 23457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:30:58.695679 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:58.695773 23464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:30:58.695796 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:30:58.696478 23457 server_base.cc:1034] running on GCE node
I20260810 19:30:58.697310 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:30:58.698678 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:30:58.699915 23457 hybrid_clock.cc:648] HybridClock initialized: now 1786390258699853 us; error 41 us; skew 500 ppm
I20260810 19:30:58.700278 23457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:30:58 dist-test-slave-2j7r krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:30:58.703472 23457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:30:58.704864 23457 webserver.cc:460] Webserver started at http://127.0.0.1:45159/ using document root <none> and password file <none>
I20260810 19:30:58.705559 23457 fs_manager.cc:359] Metadata directory not provided
I20260810 19:30:58.705672 23457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:30:58.705936 23457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:30:58.708921 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "928ff0a8ecd240c0999af8dcdd08cae6"
format_stamp: "Formatted at 2026-08-10 19:30:58 on dist-test-slave-2j7r"
server_key: "b75f3cc2fb675c7da851412b803c81a5"
server_key_iv: "dd1fc65019cdffc4d46becf676b2c7bc"
server_key_version: "encryptionkey@0"
I20260810 19:30:58.709755 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "928ff0a8ecd240c0999af8dcdd08cae6"
format_stamp: "Formatted at 2026-08-10 19:30:58 on dist-test-slave-2j7r"
server_key: "b75f3cc2fb675c7da851412b803c81a5"
server_key_iv: "dd1fc65019cdffc4d46becf676b2c7bc"
server_key_version: "encryptionkey@0"
I20260810 19:30:58.714936 23457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:30:58.718477 23485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:30:58.719669 23457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:30:58.719820 23457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "928ff0a8ecd240c0999af8dcdd08cae6"
format_stamp: "Formatted at 2026-08-10 19:30:58 on dist-test-slave-2j7r"
server_key: "b75f3cc2fb675c7da851412b803c81a5"
server_key_iv: "dd1fc65019cdffc4d46becf676b2c7bc"
server_key_version: "encryptionkey@0"
I20260810 19:30:58.719981 23457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:30:58.725631 23457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:30:58.729344 23457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:30:58.729585 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:30:58.747032 23457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37377
I20260810 19:30:58.747098 23585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37377 every 8 connection(s)
I20260810 19:30:58.748338 23457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:30:58.751794 23586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:30:58.756448 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23457
I20260810 19:30:58.756606 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:30:58.757052 19728 external_mini_cluster.cc:1428] Setting key 9d7516e8d14d7657827b6b01aa16ab8f
I20260810 19:30:58.762238 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6: Bootstrap starting.
I20260810 19:30:58.764410 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:58.758712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55221 (local address 127.0.0.1:37377)
0810 19:30:58.759468 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:58.759479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:58.759522 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:30:58.761234 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:58.761255 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:58.761554 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:58.761884 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:58.762965 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:58.762971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:58.762983 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:58.762999 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:58.763066 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:58.763085 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:58.763254 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:58.763350 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 19:30:58.763511 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":348,"threads_started":1}
I20260810 19:30:58.765738 23586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6: Neither blocks nor log segments found. Creating new log.
I20260810 19:30:58.766862 23586 log.cc:824] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:30:58.767987 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:30:58.769618 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6: No bootstrap required, opened a new log
I20260810 19:30:58.774516 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:30:58.772284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37625 (local address 127.0.0.1:37377)
0810 19:30:58.772459 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:30:58.772464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:30:58.772735 (+ 271us) server_negotiation.cc:407] Connection header received
0810 19:30:58.772891 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:30:58.772894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:30:58.772943 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:30:58.773016 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:30:58.773889 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:30:58.773893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:30:58.773896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:30:58.773912 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:30:58.773954 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:30:58.773968 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:30:58.774068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:30:58.774189 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:30:58.774240 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:30:58.777916 23586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260810 19:30:58.778244 23586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:30:58.778353 23586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928ff0a8ecd240c0999af8dcdd08cae6, State: Initialized, Role: FOLLOWER
I20260810 19:30:58.778861 23586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260810 19:30:58.779081 23586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:30:58.779206 23586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:30:58.779361 23586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:30:58.781531 23586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260810 19:30:58.781970 23586 leader_election.cc:302] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 928ff0a8ecd240c0999af8dcdd08cae6; no voters:
I20260810 19:30:58.782518 23586 leader_election.cc:288] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:30:58.782636 23607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:30:58.783237 23607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [term 1 LEADER]: Becoming Leader. State: Replica: 928ff0a8ecd240c0999af8dcdd08cae6, State: Running, Role: LEADER
I20260810 19:30:58.784061 23607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260810 19:30:58.784490 23586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:30:58.786798 23610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 928ff0a8ecd240c0999af8dcdd08cae6. Latest consensus state: current_term: 1 leader_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260810 19:30:58.787000 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ff0a8ecd240c0999af8dcdd08cae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260810 19:30:58.787249 23610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:58.787281 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:30:58.792511 23623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:30:58.796941 23623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:30:58.805301 23623 catalog_manager.cc:1347] Generated new cluster ID: c60ddd559fce4376a95e721ac7ae466f
I20260810 19:30:58.805411 23623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:30:58.827494 23623 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:30:58.827962 23623 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:30:58.845681 23623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 928ff0a8ecd240c0999af8dcdd08cae6: Generated new TSK 0
I20260810 19:30:58.846719 23623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:30:58.851858 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23457
W20260810 19:30:58.852993 23580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55221
W20260810 19:30:58.853652 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:30:58.853776 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23457
2026-08-10T19:30:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:30:58 dist-test-slave-2j7r krb5kdc[23648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:30:58 dist-test-slave-2j7r krb5kdc[23648](info): set up 2 sockets
Aug 10 19:30:58 dist-test-slave-2j7r krb5kdc[23648](info): commencing operation
krb5kdc: starting...
W20260810 19:31:00.115309 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:00 dist-test-slave-2j7r krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:00.429432 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:00.766772 23667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:00.767056 23667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:00.767194 23667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:00.774317 23667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:00.774432 23667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:00.774513 23667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:00.774592 23667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:00.774660 23667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:00.784315 23667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:00.785602 23667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:00.786896 23667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:00.798151 23678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:00.798203 23682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:00.798178 23677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:00.799026 23667 server_base.cc:1034] running on GCE node
I20260810 19:31:00.799770 23667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:00.801110 23667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:00.802412 23667 hybrid_clock.cc:648] HybridClock initialized: now 1786390260802366 us; error 26 us; skew 500 ppm
I20260810 19:31:00.802767 23667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:00 dist-test-slave-2j7r krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:00.809900 23667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:00.811604 23667 webserver.cc:460] Webserver started at http://127.0.0.1:35907/ using document root <none> and password file <none>
I20260810 19:31:00.812286 23667 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:00.812415 23667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:00.812712 23667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:00.815721 23667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "b622b8bc690c47de9c271c5d231b83ca"
format_stamp: "Formatted at 2026-08-10 19:31:00 on dist-test-slave-2j7r"
server_key: "0e76517beeec73842da3f47cf71ee397"
server_key_iv: "1a3166707dbdc4eac608fa207b273e3e"
server_key_version: "encryptionkey@0"
I20260810 19:31:00.816475 23667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "b622b8bc690c47de9c271c5d231b83ca"
format_stamp: "Formatted at 2026-08-10 19:31:00 on dist-test-slave-2j7r"
server_key: "0e76517beeec73842da3f47cf71ee397"
server_key_iv: "1a3166707dbdc4eac608fa207b273e3e"
server_key_version: "encryptionkey@0"
I20260810 19:31:00.821635 23667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 19:31:00.824923 23694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:00.826220 23667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:00.826398 23667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "b622b8bc690c47de9c271c5d231b83ca"
format_stamp: "Formatted at 2026-08-10 19:31:00 on dist-test-slave-2j7r"
server_key: "0e76517beeec73842da3f47cf71ee397"
server_key_iv: "1a3166707dbdc4eac608fa207b273e3e"
server_key_version: "encryptionkey@0"
I20260810 19:31:00.826576 23667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:00.837266 23667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:00.841285 23667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:00.841544 23667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:00.858294 23667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46767
I20260810 19:31:00.858340 23784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46767 every 8 connection(s)
I20260810 19:31:00.859616 23667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:00.861327 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23667
I20260810 19:31:00.861487 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:00.861964 19728 external_mini_cluster.cc:1428] Setting key 245c7b51c4c659ae0789de56dd34c9bd
I20260810 19:31:00.864437 23786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:31:00 dist-test-slave-2j7r krb5kdc[23648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:31:00.876933 23786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca: Bootstrap starting.
I20260810 19:31:00.880182 23786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:00.881218 23786 log.cc:824] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:00.883867 23786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca: No bootstrap required, opened a new log
I20260810 19:31:00.884804 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:00.863660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42019 (local address 127.0.0.1:46767)
0810 19:31:00.864539 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:00.864554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:00.864611 (+ 57us) server_negotiation.cc:407] Connection header received
0810 19:31:00.867008 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:00.867035 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:00.867527 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:00.869675 (+ 2148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:00.870113 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:00.871807 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:00.873113 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:00.873783 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:00.873961 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:00.877910 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:00.877918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:00.877932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:00.877966 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:00.880881 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:00.881617 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:00.881623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:00.881629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:00.881811 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:00.882220 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:00.882223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:00.882225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:00.882714 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:00.882978 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:00.883459 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 19:31:00.883836 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
I20260810 19:31:00.888489 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:00.891949 23786 raft_consensus.cc:348] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260810 19:31:00.892268 23786 raft_consensus.cc:374] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:00.892354 23786 raft_consensus.cc:729] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b622b8bc690c47de9c271c5d231b83ca, State: Initialized, Role: FOLLOWER
I20260810 19:31:00.892935 23786 consensus_queue.cc:260] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260810 19:31:00.893241 23786 raft_consensus.cc:388] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:00.893397 23786 raft_consensus.cc:482] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:00.893591 23786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:00.896324 23786 raft_consensus.cc:504] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260810 19:31:00.896876 23786 leader_election.cc:302] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b622b8bc690c47de9c271c5d231b83ca; no voters:
I20260810 19:31:00.897496 23786 leader_election.cc:288] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:00.897589 23801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:00.898228 23801 raft_consensus.cc:686] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [term 1 LEADER]: Becoming Leader. State: Replica: b622b8bc690c47de9c271c5d231b83ca, State: Running, Role: LEADER
I20260810 19:31:00.898844 23801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260810 19:31:00.900671 23786 sys_catalog.cc:564] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:00.901693 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b622b8bc690c47de9c271c5d231b83ca. Latest consensus state: current_term: 1 leader_uuid: "b622b8bc690c47de9c271c5d231b83ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260810 19:31:00.902161 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:00.905753 23804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b622b8bc690c47de9c271c5d231b83ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b622b8bc690c47de9c271c5d231b83ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260810 19:31:00.906042 23804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:00.908293 23807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:00.909430 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:00.892612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42365 (local address 127.0.0.1:46767)
0810 19:31:00.892779 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:00.892784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:00.893311 (+ 527us) server_negotiation.cc:407] Connection header received
0810 19:31:00.893503 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:00.893508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:00.893586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:00.893661 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:00.895453 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:00.896135 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:00.897863 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:00.898582 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:00.898708 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:00.901777 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:00.901783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:00.901788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:00.901834 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:00.905656 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:00.906559 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:00.906565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:00.906569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:00.906754 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:00.907208 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:00.907213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:00.907217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:00.907615 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:00.907921 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:00.908810 (+ 889us) server_negotiation.cc:299] Negotiation successful
0810 19:31:00.909033 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:31:00.913532 23807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:00.922216 23807 catalog_manager.cc:1347] Generated new cluster ID: 229ecc7be554494894d50afdfe4152b9
I20260810 19:31:00.922326 23807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:00.941129 23807 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:00.942590 23807 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:00.955785 23807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b622b8bc690c47de9c271c5d231b83ca: Generated new TSK 0
I20260810 19:31:00.956739 23807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:00.970009 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23667
I20260810 19:31:00.971269 23776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:01.062340 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23667
2026-08-10T19:31:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:01 dist-test-slave-2j7r krb5kdc[23842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:01 dist-test-slave-2j7r krb5kdc[23842](info): set up 2 sockets
Aug 10 19:31:01 dist-test-slave-2j7r krb5kdc[23842](info): commencing operation
krb5kdc: starting...
W20260810 19:31:02.327376 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:02 dist-test-slave-2j7r krb5kdc[23842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:02.646231 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:02.980697 23872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:02.980973 23872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:02.981129 23872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:02.988246 23872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:02.988349 23872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:02.988425 23872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:02.988502 23872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:02.988561 23872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:02.998147 23872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:02.999539 23872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:03.000911 23872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:03.012467 23882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:03.012653 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:03.012976 23883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:03.013954 23872 server_base.cc:1034] running on GCE node
I20260810 19:31:03.014665 23872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:03.015982 23872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:03.017176 23872 hybrid_clock.cc:648] HybridClock initialized: now 1786390263017124 us; error 41 us; skew 500 ppm
I20260810 19:31:03.017529 23872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:03 dist-test-slave-2j7r krb5kdc[23842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:03.020536 23872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:03.021941 23872 webserver.cc:460] Webserver started at http://127.0.0.1:35795/ using document root <none> and password file <none>
I20260810 19:31:03.022599 23872 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:03.022717 23872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:03.022993 23872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:03.026012 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "2f79c454c4e04d65b1366fdff7296cd7"
format_stamp: "Formatted at 2026-08-10 19:31:03 on dist-test-slave-2j7r"
server_key: "8711b5eb545cea968b7366077ba67e20"
server_key_iv: "5961466f0fae2ac0b1ff2cf3640294ec"
server_key_version: "encryptionkey@0"
I20260810 19:31:03.026770 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "2f79c454c4e04d65b1366fdff7296cd7"
format_stamp: "Formatted at 2026-08-10 19:31:03 on dist-test-slave-2j7r"
server_key: "8711b5eb545cea968b7366077ba67e20"
server_key_iv: "5961466f0fae2ac0b1ff2cf3640294ec"
server_key_version: "encryptionkey@0"
I20260810 19:31:03.031745 23872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:31:03.035315 23897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:03.036552 23872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:31:03.036716 23872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2f79c454c4e04d65b1366fdff7296cd7"
format_stamp: "Formatted at 2026-08-10 19:31:03 on dist-test-slave-2j7r"
server_key: "8711b5eb545cea968b7366077ba67e20"
server_key_iv: "5961466f0fae2ac0b1ff2cf3640294ec"
server_key_version: "encryptionkey@0"
I20260810 19:31:03.036888 23872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:03.050786 23872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:03.054872 23872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:03.055133 23872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:03.072759 23872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44051
I20260810 19:31:03.072783 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44051 every 8 connection(s)
I20260810 19:31:03.074132 23872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:03.074994 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 23872
I20260810 19:31:03.075178 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:03.075719 19728 external_mini_cluster.cc:1428] Setting key ad3b9fc17e76c0bca1594c2d518c540a
I20260810 19:31:03.078502 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:03.088660 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:03.077507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43355 (local address 127.0.0.1:44051)
0810 19:31:03.078261 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:03.078272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:03.078313 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:31:03.079957 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:03.079979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:03.080343 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:03.080631 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:03.081254 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:03.082559 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:03.084336 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:03.085068 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:03.085229 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:03.086460 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:03.086467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:03.086479 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:03.087002 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:03.087077 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:03.087101 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:03.087193 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:03.087418 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 19:31:03.087656 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":341,"threads_started":1}
I20260810 19:31:03.090615 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7: Bootstrap starting.
I20260810 19:31:03.092067 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:03.093672 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:03.094592 24005 log.cc:824] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:03.097373 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7: No bootstrap required, opened a new log
I20260810 19:31:03.102474 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:03.096178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40699 (local address 127.0.0.1:44051)
0810 19:31:03.096331 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:03.096337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:03.096796 (+ 459us) server_negotiation.cc:407] Connection header received
0810 19:31:03.096943 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:03.096947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:03.097018 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:03.097140 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:03.097632 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:03.098719 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:03.099986 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:03.100664 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:03.100798 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:03.101737 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:03.101744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:03.101749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:03.101770 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:03.101818 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:03.101838 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:03.101947 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:03.102039 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 19:31:03.102132 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:31:03.105913 24005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } }
I20260810 19:31:03.106283 24005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:03.106383 24005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f79c454c4e04d65b1366fdff7296cd7, State: Initialized, Role: FOLLOWER
I20260810 19:31:03.106890 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } }
I20260810 19:31:03.107095 24005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:03.107223 24005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:03.107386 24005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:03.109750 24005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } }
I20260810 19:31:03.110200 24005 leader_election.cc:302] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f79c454c4e04d65b1366fdff7296cd7; no voters:
I20260810 19:31:03.110725 24005 leader_election.cc:288] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:03.110873 24018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:03.111526 24018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [term 1 LEADER]: Becoming Leader. State: Replica: 2f79c454c4e04d65b1366fdff7296cd7, State: Running, Role: LEADER
I20260810 19:31:03.112051 24018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } }
I20260810 19:31:03.112690 24005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:03.114810 24023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f79c454c4e04d65b1366fdff7296cd7. Latest consensus state: current_term: 1 leader_uuid: "2f79c454c4e04d65b1366fdff7296cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } } }
I20260810 19:31:03.114758 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f79c454c4e04d65b1366fdff7296cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79c454c4e04d65b1366fdff7296cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44051 } } }
I20260810 19:31:03.115231 24023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:03.115641 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:03.117727 24033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:03.122542 24033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:03.131297 24033 catalog_manager.cc:1347] Generated new cluster ID: 84b8cd9ba735481e882eb27aabc236d5
I20260810 19:31:03.131405 24033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:03.141624 24033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:03.143002 24033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:03.156687 24033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f79c454c4e04d65b1366fdff7296cd7: Generated new TSK 0
I20260810 19:31:03.157712 24033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:03.164578 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23872
W20260810 19:31:03.165921 23993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43355
W20260810 19:31:03.166602 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 23872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:03.166759 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 23872
2026-08-10T19:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:03 dist-test-slave-2j7r krb5kdc[24058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:03 dist-test-slave-2j7r krb5kdc[24058](info): set up 2 sockets
Aug 10 19:31:03 dist-test-slave-2j7r krb5kdc[24058](info): commencing operation
krb5kdc: starting...
W20260810 19:31:04.440313 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.002s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:04 dist-test-slave-2j7r krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:04.756623 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:05.102623 24080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:05.102936 24080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:05.103081 24080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:05.110419 24080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:05.110539 24080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:05.110630 24080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:05.110736 24080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:05.110802 24080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:05.120960 24080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:05.122284 24080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:05.123620 24080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:05.135560 24088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:05.135486 24091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:05.135521 24095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:05.136374 24080 server_base.cc:1034] running on GCE node
I20260810 19:31:05.137203 24080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:05.138703 24080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:05.139950 24080 hybrid_clock.cc:648] HybridClock initialized: now 1786390265139887 us; error 62 us; skew 500 ppm
I20260810 19:31:05.140298 24080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:05 dist-test-slave-2j7r krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:05.143407 24080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:05.144814 24080 webserver.cc:460] Webserver started at http://127.0.0.1:36101/ using document root <none> and password file <none>
I20260810 19:31:05.145505 24080 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:05.145622 24080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:05.145915 24080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:05.148794 24080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "1ffc570fcc09428692aba2eff4216e41"
format_stamp: "Formatted at 2026-08-10 19:31:05 on dist-test-slave-2j7r"
server_key: "78bab8f0d35839ec605ba7f2d99ea40b"
server_key_iv: "ae4283026d102f581f5464c728399d5f"
server_key_version: "encryptionkey@0"
I20260810 19:31:05.149523 24080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "1ffc570fcc09428692aba2eff4216e41"
format_stamp: "Formatted at 2026-08-10 19:31:05 on dist-test-slave-2j7r"
server_key: "78bab8f0d35839ec605ba7f2d99ea40b"
server_key_iv: "ae4283026d102f581f5464c728399d5f"
server_key_version: "encryptionkey@0"
I20260810 19:31:05.154438 24080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:31:05.157797 24110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:05.159051 24080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:31:05.159193 24080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "1ffc570fcc09428692aba2eff4216e41"
format_stamp: "Formatted at 2026-08-10 19:31:05 on dist-test-slave-2j7r"
server_key: "78bab8f0d35839ec605ba7f2d99ea40b"
server_key_iv: "ae4283026d102f581f5464c728399d5f"
server_key_version: "encryptionkey@0"
I20260810 19:31:05.159366 24080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:05.164875 24080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:05.168571 24080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:05.168807 24080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:05.185896 24080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38829
I20260810 19:31:05.185976 24200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38829 every 8 connection(s)
I20260810 19:31:05.187201 24080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:05.190775 24207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:05.196049 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24080
I20260810 19:31:05.196208 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:05.196606 19728 external_mini_cluster.cc:1428] Setting key 529092daf97213c64a718dd8f3b48e21
I20260810 19:31:05.201050 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41: Bootstrap starting.
I20260810 19:31:05.204325 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:05.198334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43979 (local address 127.0.0.1:38829)
0810 19:31:05.199235 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:05.199247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:05.199292 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:31:05.200909 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:05.200932 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:05.201314 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:05.201615 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:05.202715 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:05.202721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:05.202737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:05.202752 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:05.202815 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:05.202859 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:05.203055 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:05.203201 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:31:05.203380 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":408,"threads_started":1}
I20260810 19:31:05.204633 24207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:05.205757 24207 log.cc:824] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:05.208077 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:05.208963 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41: No bootstrap required, opened a new log
I20260810 19:31:05.215482 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:05.213158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46517 (local address 127.0.0.1:38829)
0810 19:31:05.213336 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:05.213342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:05.213750 (+ 408us) server_negotiation.cc:407] Connection header received
0810 19:31:05.213888 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:05.213893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:05.213957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:05.214070 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:05.214851 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:05.214857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:05.214861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:05.214879 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:05.214933 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:05.214955 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:05.215081 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:05.215127 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 19:31:05.215181 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:31:05.218562 24207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } }
I20260810 19:31:05.218914 24207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:05.218999 24207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffc570fcc09428692aba2eff4216e41, State: Initialized, Role: FOLLOWER
I20260810 19:31:05.219491 24207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } }
I20260810 19:31:05.219702 24207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:05.219797 24207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:05.219924 24207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:05.222450 24207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } }
I20260810 19:31:05.222916 24207 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ffc570fcc09428692aba2eff4216e41; no voters:
I20260810 19:31:05.223464 24207 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:05.223613 24226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:05.224231 24226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffc570fcc09428692aba2eff4216e41, State: Running, Role: LEADER
I20260810 19:31:05.224736 24226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } }
I20260810 19:31:05.225512 24207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:05.228420 24229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ffc570fcc09428692aba2eff4216e41. Latest consensus state: current_term: 1 leader_uuid: "1ffc570fcc09428692aba2eff4216e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } } }
I20260810 19:31:05.229105 24229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:05.228451 24227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ffc570fcc09428692aba2eff4216e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffc570fcc09428692aba2eff4216e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38829 } } }
I20260810 19:31:05.229535 24227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:05.233724 24233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:05.238552 24233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:05.246346 24233 catalog_manager.cc:1347] Generated new cluster ID: 245bdd05a5164dec8a92e5d87eb94c05
I20260810 19:31:05.246459 24233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:05.261435 24233 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:05.261880 24233 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:05.269786 24233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ffc570fcc09428692aba2eff4216e41: Generated new TSK 0
I20260810 19:31:05.270746 24233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:05.284687 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24080
W20260810 19:31:05.285890 24195 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43979
W20260810 19:31:05.286623 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24080: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:05.286762 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24080
2026-08-10T19:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:05 dist-test-slave-2j7r krb5kdc[24257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:05 dist-test-slave-2j7r krb5kdc[24257](info): set up 2 sockets
Aug 10 19:31:05 dist-test-slave-2j7r krb5kdc[24257](info): commencing operation
krb5kdc: starting...
W20260810 19:31:06.563369 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:06 dist-test-slave-2j7r krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:06.870289 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:07.214915 24278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:07.215224 24278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:07.215327 24278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:07.222697 24278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:07.222834 24278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:07.222882 24278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:07.222941 24278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:07.222985 24278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:07.233121 24278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:07.234488 24278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:07.235736 24278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:07.247524 24285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:07.247609 24286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:07.247551 24290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:07.249058 24278 server_base.cc:1034] running on GCE node
I20260810 19:31:07.249909 24278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:07.251410 24278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:07.252629 24278 hybrid_clock.cc:648] HybridClock initialized: now 1786390267252568 us; error 45 us; skew 500 ppm
I20260810 19:31:07.252959 24278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:07 dist-test-slave-2j7r krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:07.261328 24278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:07.262946 24278 webserver.cc:460] Webserver started at http://127.0.0.1:43313/ using document root <none> and password file <none>
I20260810 19:31:07.263616 24278 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:07.263720 24278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:07.263978 24278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:07.267022 24278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "baef10be7cf74492af66ef72f785a44e"
format_stamp: "Formatted at 2026-08-10 19:31:07 on dist-test-slave-2j7r"
server_key: "4660aa073a2efaf3a0f5634980c0d949"
server_key_iv: "12cca87a5fe844fc278c5891ad0e4d7a"
server_key_version: "encryptionkey@0"
I20260810 19:31:07.267759 24278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "baef10be7cf74492af66ef72f785a44e"
format_stamp: "Formatted at 2026-08-10 19:31:07 on dist-test-slave-2j7r"
server_key: "4660aa073a2efaf3a0f5634980c0d949"
server_key_iv: "12cca87a5fe844fc278c5891ad0e4d7a"
server_key_version: "encryptionkey@0"
I20260810 19:31:07.272918 24278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:31:07.276384 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:07.277926 24278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:07.278103 24278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "baef10be7cf74492af66ef72f785a44e"
format_stamp: "Formatted at 2026-08-10 19:31:07 on dist-test-slave-2j7r"
server_key: "4660aa073a2efaf3a0f5634980c0d949"
server_key_iv: "12cca87a5fe844fc278c5891ad0e4d7a"
server_key_version: "encryptionkey@0"
I20260810 19:31:07.278342 24278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:07.309438 24278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:07.313815 24278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:07.314095 24278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:07.332299 24278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33633
I20260810 19:31:07.332360 24393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33633 every 8 connection(s)
I20260810 19:31:07.333745 24278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:07.337693 24394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:07.343050 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24278
I20260810 19:31:07.343230 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:07.343647 19728 external_mini_cluster.cc:1428] Setting key 6c4a802d1004d0d98adf4963aaeaf363
I20260810 19:31:07.349699 24394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e: Bootstrap starting.
I20260810 19:31:07.353504 24394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:07.354489 24394 log.cc:824] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:07.357293 24394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e: No bootstrap required, opened a new log
Aug 10 19:31:07 dist-test-slave-2j7r krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:31:07.367141 24394 raft_consensus.cc:348] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260810 19:31:07.367633 24394 raft_consensus.cc:374] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:07.367744 24394 raft_consensus.cc:729] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baef10be7cf74492af66ef72f785a44e, State: Initialized, Role: FOLLOWER
I20260810 19:31:07.368424 24394 consensus_queue.cc:260] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260810 19:31:07.368759 24394 raft_consensus.cc:388] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:07.368908 24394 raft_consensus.cc:482] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:07.369118 24394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:07.369216 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:07.345733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:57443 (local address 127.0.0.1:33633)
0810 19:31:07.346617 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:07.346635 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:07.346686 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:31:07.348990 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:07.349016 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:07.349565 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:07.350005 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:07.351331 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:07.353301 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:07.354573 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:07.355493 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:07.355672 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:07.360049 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:07.360065 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:07.360085 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:07.360132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:07.364231 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:07.365170 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:07.365180 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:07.365188 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:07.365413 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:07.365855 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:07.365860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:07.365863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:07.366635 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:07.366948 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:07.367358 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 19:31:07.367743 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":364,"threads_started":1}
I20260810 19:31:07.372035 24394 raft_consensus.cc:504] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260810 19:31:07.372552 24394 leader_election.cc:302] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baef10be7cf74492af66ef72f785a44e; no voters:
I20260810 19:31:07.373225 24394 leader_election.cc:288] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:07.373370 24400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:07.373421 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:07.374483 24400 raft_consensus.cc:686] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [term 1 LEADER]: Becoming Leader. State: Replica: baef10be7cf74492af66ef72f785a44e, State: Running, Role: LEADER
I20260810 19:31:07.375011 24400 consensus_queue.cc:237] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260810 19:31:07.375444 24394 sys_catalog.cc:564] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:07.377714 24401 sys_catalog.cc:455] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baef10be7cf74492af66ef72f785a44e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } } }
I20260810 19:31:07.378103 24401 sys_catalog.cc:458] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:07.379261 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [sys.catalog]: SysCatalogTable state changed. Reason: New leader baef10be7cf74492af66ef72f785a44e. Latest consensus state: current_term: 1 leader_uuid: "baef10be7cf74492af66ef72f785a44e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baef10be7cf74492af66ef72f785a44e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } } }
I20260810 19:31:07.379648 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:07.380913 24419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:07.385908 24419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:07.399950 24419 catalog_manager.cc:1347] Generated new cluster ID: 9ad7e002b59f449fbe124ea008ad1a19
I20260810 19:31:07.400069 24419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:07.414995 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:07.390057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:60599 (local address 127.0.0.1:33633)
0810 19:31:07.390249 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:07.390257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:07.397823 (+ 7566us) server_negotiation.cc:407] Connection header received
0810 19:31:07.398051 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:07.398057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:07.398143 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:07.398272 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:07.400571 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:07.401780 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:07.403550 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:07.404191 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:07.404293 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:07.407560 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:07.407566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:07.407571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:07.407606 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:07.411076 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:07.412049 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:07.412054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:07.412058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:07.412227 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:07.412731 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:07.412736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:07.412740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:07.413118 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:07.413351 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:07.414449 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0810 19:31:07.414618 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:31:07.437281 24419 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:07.439143 24419 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:07.448639 24419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baef10be7cf74492af66ef72f785a44e: Generated new TSK 0
I20260810 19:31:07.449592 24419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:07.462160 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24278
I20260810 19:31:07.463371 24384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:07.555552 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24278
W20260810 19:31:07.581313 24254 connection.cc:535] client connection to 127.0.0.1:33633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33633): Connection reset by peer (error 104)
2026-08-10T19:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:07 dist-test-slave-2j7r krb5kdc[24455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:07 dist-test-slave-2j7r krb5kdc[24455](info): set up 2 sockets
Aug 10 19:31:07 dist-test-slave-2j7r krb5kdc[24455](info): commencing operation
krb5kdc: starting...
W20260810 19:31:08.867257 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:09 dist-test-slave-2j7r krb5kdc[24455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:09.185616 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:09.520660 24474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:09.520932 24474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:09.521066 24474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:09.528280 24474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:09.528388 24474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:09.528460 24474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:09.528540 24474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:09.528594 24474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:09.538993 24474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:09.540297 24474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:09.541610 24474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:09.553036 24479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:09.552943 24480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:09.552903 24482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:09.553893 24474 server_base.cc:1034] running on GCE node
I20260810 19:31:09.554631 24474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:09.555948 24474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:09.557163 24474 hybrid_clock.cc:648] HybridClock initialized: now 1786390269557118 us; error 33 us; skew 500 ppm
I20260810 19:31:09.557503 24474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:09 dist-test-slave-2j7r krb5kdc[24455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:09.560575 24474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:09.561993 24474 webserver.cc:460] Webserver started at http://127.0.0.1:32769/ using document root <none> and password file <none>
I20260810 19:31:09.562620 24474 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:09.562724 24474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:09.562985 24474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:09.566115 24474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "b6b63957d64e4523aefb66eb95968e5b"
format_stamp: "Formatted at 2026-08-10 19:31:09 on dist-test-slave-2j7r"
server_key: "f8c0815b0c41d1aaf32c38e7533812a6"
server_key_iv: "ce41c4c45312c022fb1e4d5ac694222c"
server_key_version: "encryptionkey@0"
I20260810 19:31:09.566865 24474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "b6b63957d64e4523aefb66eb95968e5b"
format_stamp: "Formatted at 2026-08-10 19:31:09 on dist-test-slave-2j7r"
server_key: "f8c0815b0c41d1aaf32c38e7533812a6"
server_key_iv: "ce41c4c45312c022fb1e4d5ac694222c"
server_key_version: "encryptionkey@0"
I20260810 19:31:09.572788 24474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:31:09.576196 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:09.577571 24474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:09.577719 24474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "b6b63957d64e4523aefb66eb95968e5b"
format_stamp: "Formatted at 2026-08-10 19:31:09 on dist-test-slave-2j7r"
server_key: "f8c0815b0c41d1aaf32c38e7533812a6"
server_key_iv: "ce41c4c45312c022fb1e4d5ac694222c"
server_key_version: "encryptionkey@0"
I20260810 19:31:09.577900 24474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:09.610237 24474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:09.614101 24474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:09.614342 24474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:09.631538 24474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41675
I20260810 19:31:09.631604 24590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41675 every 8 connection(s)
I20260810 19:31:09.632963 24474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:09.636485 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24474
I20260810 19:31:09.636678 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:09.636838 24592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:09.637266 19728 external_mini_cluster.cc:1428] Setting key d2eaab71266bfb80d90612cd7912388c
I20260810 19:31:09.650125 24599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:09.639033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:58315 (local address 127.0.0.1:41675)
0810 19:31:09.639779 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:09.639790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:09.639841 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:31:09.641575 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:09.641598 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:09.641968 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:09.642277 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:09.642862 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:09.644151 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:09.645939 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:09.646595 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:09.646743 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:09.647983 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:09.647991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:09.648003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:09.648522 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:09.648605 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:09.648629 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:09.648742 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:09.649000 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 19:31:09.649216 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":324,"threads_started":1}
I20260810 19:31:09.650295 24592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b: Bootstrap starting.
I20260810 19:31:09.653366 24592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:09.653651 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:09.654529 24592 log.cc:824] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:09.657768 24592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b: No bootstrap required, opened a new log
I20260810 19:31:09.665213 24599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:09.658859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:45591 (local address 127.0.0.1:41675)
0810 19:31:09.659034 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:09.659039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:09.659371 (+ 332us) server_negotiation.cc:407] Connection header received
0810 19:31:09.659546 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:09.659549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:09.659609 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:09.659689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:09.660226 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:09.660985 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:09.662864 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:09.663307 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:09.663409 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:09.664505 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:09.664510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:09.664516 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:09.664532 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:09.664584 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:09.664597 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:09.664676 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:09.664847 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 19:31:09.664891 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:31:09.667119 24592 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } }
I20260810 19:31:09.667444 24592 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:09.667541 24592 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b63957d64e4523aefb66eb95968e5b, State: Initialized, Role: FOLLOWER
I20260810 19:31:09.668102 24592 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } }
I20260810 19:31:09.668326 24592 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:09.668452 24592 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:09.668596 24592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:09.670956 24592 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } }
I20260810 19:31:09.671414 24592 leader_election.cc:302] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b63957d64e4523aefb66eb95968e5b; no voters:
I20260810 19:31:09.671953 24592 leader_election.cc:288] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:09.672070 24613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:09.672704 24613 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [term 1 LEADER]: Becoming Leader. State: Replica: b6b63957d64e4523aefb66eb95968e5b, State: Running, Role: LEADER
I20260810 19:31:09.673256 24613 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } }
I20260810 19:31:09.673966 24592 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:09.675807 24614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b63957d64e4523aefb66eb95968e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } } }
I20260810 19:31:09.675841 24615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b63957d64e4523aefb66eb95968e5b. Latest consensus state: current_term: 1 leader_uuid: "b6b63957d64e4523aefb66eb95968e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b63957d64e4523aefb66eb95968e5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41675 } } }
I20260810 19:31:09.676218 24615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:09.676218 24614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:09.681171 24621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:09.687538 24621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:09.696628 24621 catalog_manager.cc:1347] Generated new cluster ID: d93463eac7dc4a2592fb7d8ce7eb061b
I20260810 19:31:09.696727 24621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:09.740530 24621 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:09.741879 24621 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:09.751500 24621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6b63957d64e4523aefb66eb95968e5b: Generated new TSK 0
I20260810 19:31:09.752269 24621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:09.766060 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24474
W20260810 19:31:09.767203 24580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.230:58315
W20260810 19:31:09.767903 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:09.768034 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24474
W20260810 19:31:09.788511 24451 connection.cc:535] client connection to 127.0.0.1:41675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41675): Connection reset by peer (error 104)
2026-08-10T19:31:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:09 dist-test-slave-2j7r krb5kdc[24655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:09 dist-test-slave-2j7r krb5kdc[24655](info): set up 2 sockets
Aug 10 19:31:09 dist-test-slave-2j7r krb5kdc[24655](info): commencing operation
krb5kdc: starting...
W20260810 19:31:11.031559 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:11 dist-test-slave-2j7r krb5kdc[24655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:11.348091 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:11.682633 24676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:11.682934 24676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:11.683086 24676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:11.690332 24676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:11.690447 24676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:11.690538 24676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:11.690636 24676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:11.690696 24676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:11.700642 24676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:11.702042 24676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:11.703538 24676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:11.715145 24686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:11.715178 24682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:11.715210 24681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:11.716429 24676 server_base.cc:1034] running on GCE node
I20260810 19:31:11.717200 24676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:11.718549 24676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:11.719787 24676 hybrid_clock.cc:648] HybridClock initialized: now 1786390271719738 us; error 27 us; skew 500 ppm
I20260810 19:31:11.720116 24676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:11 dist-test-slave-2j7r krb5kdc[24655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:11.723161 24676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:11.724537 24676 webserver.cc:460] Webserver started at http://127.0.0.1:35787/ using document root <none> and password file <none>
I20260810 19:31:11.725260 24676 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:11.725386 24676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:11.725688 24676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:11.728608 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "6262fa9033d84ac1b0e73a6bfd67af87"
format_stamp: "Formatted at 2026-08-10 19:31:11 on dist-test-slave-2j7r"
server_key: "255739096f77a3f1d7542f1975ff7c65"
server_key_iv: "be2dd069609715878a9c8f7f25847b8e"
server_key_version: "encryptionkey@0"
I20260810 19:31:11.729399 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "6262fa9033d84ac1b0e73a6bfd67af87"
format_stamp: "Formatted at 2026-08-10 19:31:11 on dist-test-slave-2j7r"
server_key: "255739096f77a3f1d7542f1975ff7c65"
server_key_iv: "be2dd069609715878a9c8f7f25847b8e"
server_key_version: "encryptionkey@0"
I20260810 19:31:11.734388 24676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 19:31:11.737812 24696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:11.739086 24676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:31:11.739255 24676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "6262fa9033d84ac1b0e73a6bfd67af87"
format_stamp: "Formatted at 2026-08-10 19:31:11 on dist-test-slave-2j7r"
server_key: "255739096f77a3f1d7542f1975ff7c65"
server_key_iv: "be2dd069609715878a9c8f7f25847b8e"
server_key_version: "encryptionkey@0"
I20260810 19:31:11.739441 24676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:11.744796 24676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:11.748566 24676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:11.748833 24676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:11.766209 24676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41289
I20260810 19:31:11.766364 24791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41289 every 8 connection(s)
I20260810 19:31:11.767494 24676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:11.771296 24793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:11.776757 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24676
I20260810 19:31:11.776949 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:11.777552 19728 external_mini_cluster.cc:1428] Setting key 0f7d1323455d89dbfd7e05335fd5564f
I20260810 19:31:11.782382 24793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87: Bootstrap starting.
I20260810 19:31:11.785274 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:11.779414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:45693 (local address 127.0.0.1:41289)
0810 19:31:11.780291 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:11.780302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:11.780362 (+ 60us) server_negotiation.cc:407] Connection header received
0810 19:31:11.782035 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:11.782056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:11.782375 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:11.782730 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:11.783728 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:11.783735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:11.783746 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:11.783764 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:11.783831 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:11.783861 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:11.784058 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:11.784199 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 19:31:11.784391 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":371,"threads_started":1}
I20260810 19:31:11.785622 24793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:11.786670 24793 log.cc:824] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:11.788712 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:11.789484 24793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87: No bootstrap required, opened a new log
I20260810 19:31:11.795296 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:11.792924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:39131 (local address 127.0.0.1:41289)
0810 19:31:11.793122 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:11.793128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:11.793437 (+ 309us) server_negotiation.cc:407] Connection header received
0810 19:31:11.793570 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:11.793579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:11.793635 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:11.793739 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:11.794612 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:11.794618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:11.794623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:11.794641 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:11.794685 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:11.794705 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:11.794832 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:11.794954 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 19:31:11.795015 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:31:11.797528 24793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } }
I20260810 19:31:11.797853 24793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:11.797925 24793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6262fa9033d84ac1b0e73a6bfd67af87, State: Initialized, Role: FOLLOWER
I20260810 19:31:11.798429 24793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } }
I20260810 19:31:11.798637 24793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:11.798755 24793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:11.798900 24793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:11.801017 24793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } }
I20260810 19:31:11.801535 24793 leader_election.cc:302] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6262fa9033d84ac1b0e73a6bfd67af87; no voters:
I20260810 19:31:11.802057 24793 leader_election.cc:288] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:11.802175 24807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:11.802776 24807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [term 1 LEADER]: Becoming Leader. State: Replica: 6262fa9033d84ac1b0e73a6bfd67af87, State: Running, Role: LEADER
I20260810 19:31:11.803296 24807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } }
I20260810 19:31:11.803938 24793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:11.807076 24810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6262fa9033d84ac1b0e73a6bfd67af87. Latest consensus state: current_term: 1 leader_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } } }
I20260810 19:31:11.807507 24810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:11.808008 24808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6262fa9033d84ac1b0e73a6bfd67af87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41289 } } }
I20260810 19:31:11.808295 24808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:11.808815 24819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:11.813218 24819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:11.821298 24819 catalog_manager.cc:1347] Generated new cluster ID: 19ee1d202c9d4e2d8737998e6e0c94f2
I20260810 19:31:11.821393 24819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:11.855574 24819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:11.856058 24819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:11.876977 24819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6262fa9033d84ac1b0e73a6bfd67af87: Generated new TSK 0
I20260810 19:31:11.878046 24819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:11.886958 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24676
W20260810 19:31:11.888190 24783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.230:45693
W20260810 19:31:11.888926 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:11.889081 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24676
2026-08-10T19:31:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:12 dist-test-slave-2j7r krb5kdc[24842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:12 dist-test-slave-2j7r krb5kdc[24842](info): set up 2 sockets
Aug 10 19:31:12 dist-test-slave-2j7r krb5kdc[24842](info): commencing operation
krb5kdc: starting...
W20260810 19:31:13.160605 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:13 dist-test-slave-2j7r krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:13.473994 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:13.810889 24864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:13.811173 24864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:13.811311 24864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:13.818331 24864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:13.818439 24864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:13.818517 24864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:13.818598 24864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:13.818655 24864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:13.828222 24864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:13.829499 24864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:13.830792 24864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:13.842088 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:13.842044 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:13.842114 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:13.842886 24864 server_base.cc:1034] running on GCE node
I20260810 19:31:13.843647 24864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:13.844985 24864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:13.846213 24864 hybrid_clock.cc:648] HybridClock initialized: now 1786390273846174 us; error 44 us; skew 500 ppm
I20260810 19:31:13.846585 24864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:13 dist-test-slave-2j7r krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:13.854140 24864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:13.855654 24864 webserver.cc:460] Webserver started at http://127.0.0.1:38943/ using document root <none> and password file <none>
I20260810 19:31:13.856308 24864 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:13.856437 24864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:13.856727 24864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:13.859692 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "29fd86efc0b04bb1a3f95004d6f73570"
format_stamp: "Formatted at 2026-08-10 19:31:13 on dist-test-slave-2j7r"
server_key: "947865cc698e4cb44a4b5cd5391e030b"
server_key_iv: "c191e90d33dafce3d7301b6d08b4546b"
server_key_version: "encryptionkey@0"
I20260810 19:31:13.860452 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "29fd86efc0b04bb1a3f95004d6f73570"
format_stamp: "Formatted at 2026-08-10 19:31:13 on dist-test-slave-2j7r"
server_key: "947865cc698e4cb44a4b5cd5391e030b"
server_key_iv: "c191e90d33dafce3d7301b6d08b4546b"
server_key_version: "encryptionkey@0"
I20260810 19:31:13.865468 24864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:31:13.868713 24881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:13.870008 24864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:13.870198 24864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "29fd86efc0b04bb1a3f95004d6f73570"
format_stamp: "Formatted at 2026-08-10 19:31:13 on dist-test-slave-2j7r"
server_key: "947865cc698e4cb44a4b5cd5391e030b"
server_key_iv: "c191e90d33dafce3d7301b6d08b4546b"
server_key_version: "encryptionkey@0"
I20260810 19:31:13.870373 24864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:13.895699 24864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:13.899682 24864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:13.899950 24864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:13.918102 24864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43435
I20260810 19:31:13.918133 24975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43435 every 8 connection(s)
I20260810 19:31:13.919461 24864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:13.923218 24976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:13.927922 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 24864
I20260810 19:31:13.928114 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:13.928699 19728 external_mini_cluster.cc:1428] Setting key be524fe643a4669e606176ff13342921
I20260810 19:31:13.934388 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570: Bootstrap starting.
I20260810 19:31:13.938026 24976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:13.939200 24976 log.cc:824] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:13.941983 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570: No bootstrap required, opened a new log
Aug 10 19:31:13 dist-test-slave-2j7r krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:31:13.952023 24976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260810 19:31:13.952492 24976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:13.952610 24976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fd86efc0b04bb1a3f95004d6f73570, State: Initialized, Role: FOLLOWER
I20260810 19:31:13.952980 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:13.930534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:36209 (local address 127.0.0.1:43435)
0810 19:31:13.931305 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:13.931317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:13.931362 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:31:13.933193 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:13.933214 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:13.933594 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:13.933894 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:13.935750 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:13.937081 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:13.938834 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:13.939543 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:13.939710 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:13.943647 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:13.943658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:13.943676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:13.943733 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:13.947924 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:13.948792 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:13.948802 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:13.948810 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:13.949045 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:13.949582 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:13.949588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:13.949591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:13.950380 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:13.950692 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:13.951178 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 19:31:13.951523 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260810 19:31:13.953323 24976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260810 19:31:13.953636 24976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:13.953776 24976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:13.953979 24976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:13.956658 24976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260810 19:31:13.956885 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:13.957185 24976 leader_election.cc:302] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29fd86efc0b04bb1a3f95004d6f73570; no voters:
I20260810 19:31:13.957798 24976 leader_election.cc:288] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:13.957957 24984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:13.959669 24976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:13.962225 24984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [term 1 LEADER]: Becoming Leader. State: Replica: 29fd86efc0b04bb1a3f95004d6f73570, State: Running, Role: LEADER
I20260810 19:31:13.962905 24984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260810 19:31:13.966066 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29fd86efc0b04bb1a3f95004d6f73570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260810 19:31:13.966089 24986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29fd86efc0b04bb1a3f95004d6f73570. Latest consensus state: current_term: 1 leader_uuid: "29fd86efc0b04bb1a3f95004d6f73570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd86efc0b04bb1a3f95004d6f73570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260810 19:31:13.966547 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:13.966574 24986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:13.977802 24998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:13.980538 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:13.962852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:54569 (local address 127.0.0.1:43435)
0810 19:31:13.963114 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:13.963120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:13.963141 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:31:13.963358 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:13.963364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:13.963442 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:13.963510 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:13.965559 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:13.966777 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:13.968833 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:13.969575 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:13.969671 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:13.973192 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:13.973199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:13.973205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:13.973257 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:13.976997 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:13.978084 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:13.978089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:13.978093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:13.978228 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:13.978711 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:13.978715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:13.978718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:13.979140 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:13.979425 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:13.979900 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 19:31:13.980129 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 19:31:13.983059 24998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:13.992348 24998 catalog_manager.cc:1347] Generated new cluster ID: 58497cf56fb54462a7b0fe1592984bd0
I20260810 19:31:13.992449 24998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:14.003278 24998 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:14.004522 24998 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:14.014170 24998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29fd86efc0b04bb1a3f95004d6f73570: Generated new TSK 0
I20260810 19:31:14.015185 24998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:14.025892 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 24864
I20260810 19:31:14.027232 24965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:14.115723 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 24864
W20260810 19:31:14.138291 24838 connection.cc:535] client connection to 127.0.0.1:43435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43435): Connection reset by peer (error 104)
2026-08-10T19:31:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:14 dist-test-slave-2j7r krb5kdc[25021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:14 dist-test-slave-2j7r krb5kdc[25021](info): set up 2 sockets
Aug 10 19:31:14 dist-test-slave-2j7r krb5kdc[25021](info): commencing operation
krb5kdc: starting...
W20260810 19:31:15.390784 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.011s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:15 dist-test-slave-2j7r krb5kdc[25021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:15.714355 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:16.066294 25042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:16.066591 25042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:16.066749 25042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:16.074129 25042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:16.074278 25042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:16.074365 25042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:16.074441 25042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:16.074491 25042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:16.084321 25042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:16.085589 25042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:16.086925 25042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:16.098766 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:16.098704 25052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:16.098675 25058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:16.099699 25042 server_base.cc:1034] running on GCE node
I20260810 19:31:16.100523 25042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:16.102011 25042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:16.103283 25042 hybrid_clock.cc:648] HybridClock initialized: now 1786390276103190 us; error 41 us; skew 500 ppm
I20260810 19:31:16.103619 25042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:16 dist-test-slave-2j7r krb5kdc[25021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:16.106979 25042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:16.108428 25042 webserver.cc:460] Webserver started at http://127.0.0.1:37993/ using document root <none> and password file <none>
I20260810 19:31:16.109055 25042 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:16.109201 25042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:16.109510 25042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:16.112541 25042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "d4547524f910478982fa32c8c38d5aa5"
format_stamp: "Formatted at 2026-08-10 19:31:16 on dist-test-slave-2j7r"
server_key: "aadf42896bd9d09f70a44bf7ac0eefa1"
server_key_iv: "bbf80e4eb6145eece76b89299d8b41c9"
server_key_version: "encryptionkey@0"
I20260810 19:31:16.113318 25042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "d4547524f910478982fa32c8c38d5aa5"
format_stamp: "Formatted at 2026-08-10 19:31:16 on dist-test-slave-2j7r"
server_key: "aadf42896bd9d09f70a44bf7ac0eefa1"
server_key_iv: "bbf80e4eb6145eece76b89299d8b41c9"
server_key_version: "encryptionkey@0"
I20260810 19:31:16.118242 25042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:31:16.121513 25067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:16.122757 25042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:31:16.122924 25042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "d4547524f910478982fa32c8c38d5aa5"
format_stamp: "Formatted at 2026-08-10 19:31:16 on dist-test-slave-2j7r"
server_key: "aadf42896bd9d09f70a44bf7ac0eefa1"
server_key_iv: "bbf80e4eb6145eece76b89299d8b41c9"
server_key_version: "encryptionkey@0"
I20260810 19:31:16.123096 25042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:16.134757 25042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:16.138644 25042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:16.138914 25042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:16.156697 25042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40965
I20260810 19:31:16.156720 25162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40965 every 8 connection(s)
I20260810 19:31:16.158066 25042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:16.161701 25164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:16.166098 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25042
I20260810 19:31:16.166304 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:16.166725 19728 external_mini_cluster.cc:1428] Setting key 80f568a341f3fab55a8e61dd8624c58b
I20260810 19:31:16.172470 25164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5: Bootstrap starting.
I20260810 19:31:16.175946 25164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:16.177213 25164 log.cc:824] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:16.179867 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:16.168388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:57227 (local address 127.0.0.1:40965)
0810 19:31:16.169138 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:16.169148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:16.169193 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:31:16.171004 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:16.171026 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:16.171381 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:16.171682 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:16.172272 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:16.173658 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:16.175489 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:16.176175 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:16.176320 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:16.177534 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:16.177541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:16.177553 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:16.178217 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:16.178325 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:16.178358 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:16.178537 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:16.178755 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 19:31:16.178920 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260810 19:31:16.180173 25164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5: No bootstrap required, opened a new log
I20260810 19:31:16.183441 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:16.189459 25164 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } }
I20260810 19:31:16.189869 25164 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:16.189970 25164 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4547524f910478982fa32c8c38d5aa5, State: Initialized, Role: FOLLOWER
I20260810 19:31:16.190608 25164 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } }
I20260810 19:31:16.190897 25164 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:16.191038 25164 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:16.191236 25164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:16.193907 25164 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } }
I20260810 19:31:16.194528 25164 leader_election.cc:302] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4547524f910478982fa32c8c38d5aa5; no voters:
I20260810 19:31:16.194880 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:16.188631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:40083 (local address 127.0.0.1:40965)
0810 19:31:16.188804 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:16.188809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:16.189233 (+ 424us) server_negotiation.cc:407] Connection header received
0810 19:31:16.189380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:16.189385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:16.189474 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:16.189583 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:16.190044 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:16.191116 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:16.192393 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:16.193101 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:16.193223 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:16.194167 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:16.194174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:16.194179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:16.194201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:16.194246 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:16.194268 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:16.194386 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:16.194526 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 19:31:16.194578 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:31:16.195232 25164 leader_election.cc:288] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:16.195328 25180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:16.196015 25180 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [term 1 LEADER]: Becoming Leader. State: Replica: d4547524f910478982fa32c8c38d5aa5, State: Running, Role: LEADER
I20260810 19:31:16.196574 25180 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } }
I20260810 19:31:16.197505 25164 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:16.199504 25182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4547524f910478982fa32c8c38d5aa5. Latest consensus state: current_term: 1 leader_uuid: "d4547524f910478982fa32c8c38d5aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } } }
I20260810 19:31:16.199601 25184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4547524f910478982fa32c8c38d5aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4547524f910478982fa32c8c38d5aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40965 } } }
I20260810 19:31:16.199920 25182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:16.199954 25184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:16.202237 25188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:16.207504 25188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:16.215621 25188 catalog_manager.cc:1347] Generated new cluster ID: 54664460b8eb4ca8a17d75772ed5bee8
I20260810 19:31:16.215734 25188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:16.230068 25188 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:16.231896 25188 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:16.242432 25188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4547524f910478982fa32c8c38d5aa5: Generated new TSK 0
I20260810 19:31:16.243539 25188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:16.251196 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25042
W20260810 19:31:16.252435 25153 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.230:57227
W20260810 19:31:16.253340 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25042: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:16.253525 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25042
W20260810 19:31:16.280460 25016 connection.cc:535] client connection to 127.0.0.1:40965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40965): Connection reset by peer (error 104)
2026-08-10T19:31:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:16 dist-test-slave-2j7r krb5kdc[25212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:16 dist-test-slave-2j7r krb5kdc[25212](info): set up 2 sockets
Aug 10 19:31:16 dist-test-slave-2j7r krb5kdc[25212](info): commencing operation
krb5kdc: starting...
W20260810 19:31:17.529834 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:17 dist-test-slave-2j7r krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:31:17.858134 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:18.191004 25229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:18.191280 25229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:18.191380 25229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:18.198401 25229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:18.198520 25229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:18.198575 25229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:18.198635 25229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:18.198679 25229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:18.208866 25229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:18.210212 25229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:18.211520 25229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:18.223210 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:18.223227 25242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:18.223933 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:18.223906 25229 server_base.cc:1034] running on GCE node
I20260810 19:31:18.224781 25229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:18.226365 25229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:18.227572 25229 hybrid_clock.cc:648] HybridClock initialized: now 1786390278227527 us; error 40 us; skew 500 ppm
I20260810 19:31:18.227916 25229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:18 dist-test-slave-2j7r krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:18.231007 25229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:31:18.232446 25229 webserver.cc:460] Webserver started at http://127.0.0.1:40305/ using document root <none> and password file <none>
I20260810 19:31:18.233140 25229 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:18.233264 25229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:18.233562 25229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:18.236572 25229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "710fb8ba5033490c815d0d7cb94cdab3"
format_stamp: "Formatted at 2026-08-10 19:31:18 on dist-test-slave-2j7r"
server_key: "f702bf9141cc96e611e759373c47292b"
server_key_iv: "5394fe287de184ea3ac50f9e2b10b6e3"
server_key_version: "encryptionkey@0"
I20260810 19:31:18.237367 25229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "710fb8ba5033490c815d0d7cb94cdab3"
format_stamp: "Formatted at 2026-08-10 19:31:18 on dist-test-slave-2j7r"
server_key: "f702bf9141cc96e611e759373c47292b"
server_key_iv: "5394fe287de184ea3ac50f9e2b10b6e3"
server_key_version: "encryptionkey@0"
I20260810 19:31:18.243654 25229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:31:18.247144 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:18.248432 25229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:31:18.248590 25229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "710fb8ba5033490c815d0d7cb94cdab3"
format_stamp: "Formatted at 2026-08-10 19:31:18 on dist-test-slave-2j7r"
server_key: "f702bf9141cc96e611e759373c47292b"
server_key_iv: "5394fe287de184ea3ac50f9e2b10b6e3"
server_key_version: "encryptionkey@0"
I20260810 19:31:18.248769 25229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:18.254148 25229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:18.257903 25229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:18.258160 25229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:18.275081 25229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38419
I20260810 19:31:18.275123 25334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38419 every 8 connection(s)
I20260810 19:31:18.276396 25229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:18.277197 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25229
I20260810 19:31:18.277367 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:18.277904 19728 external_mini_cluster.cc:1428] Setting key dd2895bb6be6bccc3bcd731d166d0301
I20260810 19:31:18.280651 25336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:18.285318 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:18.279629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:45377 (local address 127.0.0.1:38419)
0810 19:31:18.280355 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:18.280366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:18.280408 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:31:18.282038 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:18.282059 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:18.282364 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:18.282706 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:18.283691 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:18.283697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:18.283709 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:18.283723 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:18.283796 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:18.283818 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:18.284044 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:18.284186 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 19:31:18.284357 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":313,"threads_started":1}
I20260810 19:31:18.288866 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:18.291814 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3: Bootstrap starting.
I20260810 19:31:18.294924 25336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:18.295310 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:18.293192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.230:43123 (local address 127.0.0.1:38419)
0810 19:31:18.293357 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:18.293363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:18.293577 (+ 214us) server_negotiation.cc:407] Connection header received
0810 19:31:18.293747 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:18.293751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:18.293795 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:18.293872 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:18.294715 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:18.294721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:18.294725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:18.294742 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:18.294778 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:18.294792 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:18.294893 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:18.294993 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 19:31:18.295050 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:31:18.296041 25336 log.cc:824] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:18.298601 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3: No bootstrap required, opened a new log
I20260810 19:31:18.305651 25336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260810 19:31:18.306007 25336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:18.306093 25336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710fb8ba5033490c815d0d7cb94cdab3, State: Initialized, Role: FOLLOWER
I20260810 19:31:18.306665 25336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260810 19:31:18.306910 25336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:18.307035 25336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:18.307188 25336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:18.309247 25336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260810 19:31:18.309759 25336 leader_election.cc:302] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710fb8ba5033490c815d0d7cb94cdab3; no voters:
I20260810 19:31:18.310290 25336 leader_election.cc:288] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:18.310401 25357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:18.310998 25357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [term 1 LEADER]: Becoming Leader. State: Replica: 710fb8ba5033490c815d0d7cb94cdab3, State: Running, Role: LEADER
I20260810 19:31:18.311467 25357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260810 19:31:18.312136 25336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:18.314301 25359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710fb8ba5033490c815d0d7cb94cdab3. Latest consensus state: current_term: 1 leader_uuid: "710fb8ba5033490c815d0d7cb94cdab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } } }
I20260810 19:31:18.314653 25359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:18.315343 25358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710fb8ba5033490c815d0d7cb94cdab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710fb8ba5033490c815d0d7cb94cdab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } } }
I20260810 19:31:18.315743 25358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:18.317930 25370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:18.323400 25370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:18.331701 25370 catalog_manager.cc:1347] Generated new cluster ID: 8d8c835dbee541be957b7065a184fa9e
I20260810 19:31:18.331815 25370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:18.366199 25370 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:18.366711 25370 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:18.376164 25370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710fb8ba5033490c815d0d7cb94cdab3: Generated new TSK 0
I20260810 19:31:18.376982 25370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:18.393890 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25229
W20260810 19:31:18.395017 25330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.230:45377
W20260810 19:31:18.395695 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:18.395829 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25229
2026-08-10T19:31:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2138 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:18 dist-test-slave-2j7r krb5kdc[25390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:18 dist-test-slave-2j7r krb5kdc[25390](info): set up 2 sockets
Aug 10 19:31:18 dist-test-slave-2j7r krb5kdc[25390](info): commencing operation
krb5kdc: starting...
W20260810 19:31:19.661727 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.006s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:19 dist-test-slave-2j7r krb5kdc[25390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:19.977970 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33141
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:20.312595 25410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:20.312871 25410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:20.312990 25410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:20.320098 25410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:20.320199 25410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:20.320272 25410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:20.320366 25410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:20.320421 25410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:20.329881 25410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:33141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:20.331136 25410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:20.332540 25410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:20.344018 25421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:20.344074 25419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:20.344097 25418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:20.344766 25410 server_base.cc:1034] running on GCE node
I20260810 19:31:20.345544 25410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:20.346881 25410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:20.348093 25410 hybrid_clock.cc:648] HybridClock initialized: now 1786390280348043 us; error 40 us; skew 500 ppm
I20260810 19:31:20.348439 25410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:20 dist-test-slave-2j7r krb5kdc[25390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390280, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:20.355713 25410 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:20.357122 25410 webserver.cc:460] Webserver started at http://127.19.68.62:43825/ using document root <none> and password file <none>
I20260810 19:31:20.357829 25410 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:20.357934 25410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:20.358170 25410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:20.361030 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "c99d59d2913d47eea5d36cf52ca42378"
format_stamp: "Formatted at 2026-08-10 19:31:20 on dist-test-slave-2j7r"
server_key: "40b87f1f3939e1338c01c75b5431429f"
server_key_iv: "2aa8bcc4f8d1bf8be0ffc39e9660f347"
server_key_version: "encryptionkey@0"
I20260810 19:31:20.361840 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "c99d59d2913d47eea5d36cf52ca42378"
format_stamp: "Formatted at 2026-08-10 19:31:20 on dist-test-slave-2j7r"
server_key: "40b87f1f3939e1338c01c75b5431429f"
server_key_iv: "2aa8bcc4f8d1bf8be0ffc39e9660f347"
server_key_version: "encryptionkey@0"
I20260810 19:31:20.366796 25410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:31:20.370121 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:20.371362 25410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:31:20.371502 25410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "c99d59d2913d47eea5d36cf52ca42378"
format_stamp: "Formatted at 2026-08-10 19:31:20 on dist-test-slave-2j7r"
server_key: "40b87f1f3939e1338c01c75b5431429f"
server_key_iv: "2aa8bcc4f8d1bf8be0ffc39e9660f347"
server_key_version: "encryptionkey@0"
I20260810 19:31:20.371656 25410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:20.384828 25410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:20.388564 25410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:20.388790 25410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:20.405753 25410 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:33141
I20260810 19:31:20.405828 25517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:33141 every 8 connection(s)
I20260810 19:31:20.407151 25410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:20.408789 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25410
I20260810 19:31:20.408995 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:20.409507 19728 external_mini_cluster.cc:1428] Setting key 6a9255351313cb19a62bed717e1b68b5
I20260810 19:31:20.412520 25518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:31:20 dist-test-slave-2j7r krb5kdc[25390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:20.424842 25518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378: Bootstrap starting.
I20260810 19:31:20.428915 25518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:20.430152 25518 log.cc:824] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:20.433420 25518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378: No bootstrap required, opened a new log
I20260810 19:31:20.433655 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:20.411268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37985 (local address 127.19.68.62:33141)
0810 19:31:20.412067 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:20.412083 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:20.412136 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:31:20.414421 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:20.414458 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:20.414943 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:20.415379 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:20.416658 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:20.418747 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:20.420063 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:20.421076 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:20.421280 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:20.425248 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:20.425259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:20.425277 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:20.425326 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:20.429139 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:20.429918 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:20.429931 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:20.429939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:20.430145 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:20.430621 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:20.430627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:20.430631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:20.431240 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:20.431527 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:20.431987 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 19:31:20.432318 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":311,"threads_started":1}
I20260810 19:31:20.437660 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:20.441922 25518 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } }
I20260810 19:31:20.442394 25518 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:20.442498 25518 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99d59d2913d47eea5d36cf52ca42378, State: Initialized, Role: FOLLOWER
I20260810 19:31:20.443150 25518 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } }
I20260810 19:31:20.443431 25518 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:20.443583 25518 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:20.443760 25518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:20.446707 25518 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } }
I20260810 19:31:20.447283 25518 leader_election.cc:302] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99d59d2913d47eea5d36cf52ca42378; no voters:
I20260810 19:31:20.447997 25518 leader_election.cc:288] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:20.448102 25533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:20.448789 25533 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [term 1 LEADER]: Becoming Leader. State: Replica: c99d59d2913d47eea5d36cf52ca42378, State: Running, Role: LEADER
I20260810 19:31:20.449410 25533 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } }
I20260810 19:31:20.452749 25518 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:20.451063 25533 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99d59d2913d47eea5d36cf52ca42378. Latest consensus state: current_term: 1 leader_uuid: "c99d59d2913d47eea5d36cf52ca42378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } } }
I20260810 19:31:20.453815 25533 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:20.454689 25536 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99d59d2913d47eea5d36cf52ca42378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99d59d2913d47eea5d36cf52ca42378" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 33141 } } }
I20260810 19:31:20.455078 25536 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:20.457113 25544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:20.458629 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:20.441667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33243 (local address 127.19.68.62:33141)
0810 19:31:20.441803 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:20.441807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:20.442359 (+ 552us) server_negotiation.cc:407] Connection header received
0810 19:31:20.442507 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:20.442511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:20.442568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:20.442658 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:20.444412 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:20.445309 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:20.447017 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:20.447504 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:20.447594 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:20.450931 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:20.450937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:20.450943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:20.450987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:20.455606 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:20.456540 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:20.456544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:20.456547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:20.456695 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:20.457176 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:20.457181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:20.457184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:20.457499 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:20.457773 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:20.458155 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 19:31:20.458330 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:31:20.463048 25544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:20.473335 25544 catalog_manager.cc:1347] Generated new cluster ID: 889c03416293466488f42729a4d5edac
I20260810 19:31:20.473443 25544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:20.498679 25544 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:20.500027 25544 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:20.511484 25544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99d59d2913d47eea5d36cf52ca42378: Generated new TSK 0
I20260810 19:31:20.512295 25544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:20.522039 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25410
I20260810 19:31:20.523511 25511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:20.617650 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25410
W20260810 19:31:20.638402 25387 connection.cc:535] client connection to 127.19.68.62:33141 recv error: Network error: failed to read from TLS socket (remote: 127.19.68.62:33141): Connection reset by peer (error 104)
2026-08-10T19:31:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:20 dist-test-slave-2j7r krb5kdc[25571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:20 dist-test-slave-2j7r krb5kdc[25571](info): set up 2 sockets
Aug 10 19:31:20 dist-test-slave-2j7r krb5kdc[25571](info): commencing operation
krb5kdc: starting...
W20260810 19:31:21.891567 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:22 dist-test-slave-2j7r krb5kdc[25571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:22.254107 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36919
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:22.589511 25596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:22.589789 25596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:22.589942 25596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:22.596891 25596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:22.597015 25596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:22.597112 25596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:22.597198 25596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:22.597260 25596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:22.606962 25596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:36919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:22.608263 25596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:22.609671 25596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:22.621886 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:22.621928 25611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:22.621959 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:22.622663 25596 server_base.cc:1034] running on GCE node
I20260810 19:31:22.623378 25596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:22.624689 25596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:22.625882 25596 hybrid_clock.cc:648] HybridClock initialized: now 1786390282625839 us; error 38 us; skew 500 ppm
I20260810 19:31:22.626216 25596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:22 dist-test-slave-2j7r krb5kdc[25571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390282, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:22.629174 25596 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:22.630620 25596 webserver.cc:460] Webserver started at http://127.19.68.62:46321/ using document root <none> and password file <none>
I20260810 19:31:22.631256 25596 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:22.631376 25596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:22.631659 25596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:22.634573 25596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "a4bb1b83a8fc45e58139794004c727e0"
format_stamp: "Formatted at 2026-08-10 19:31:22 on dist-test-slave-2j7r"
server_key: "4bff8a7f67c54ac9a1e51563140a354d"
server_key_iv: "3da66f6d6e5188eb63e61fc771929a1b"
server_key_version: "encryptionkey@0"
I20260810 19:31:22.635294 25596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "a4bb1b83a8fc45e58139794004c727e0"
format_stamp: "Formatted at 2026-08-10 19:31:22 on dist-test-slave-2j7r"
server_key: "4bff8a7f67c54ac9a1e51563140a354d"
server_key_iv: "3da66f6d6e5188eb63e61fc771929a1b"
server_key_version: "encryptionkey@0"
I20260810 19:31:22.640213 25596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:31:22.643489 25627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:22.644701 25596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:31:22.644855 25596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "a4bb1b83a8fc45e58139794004c727e0"
format_stamp: "Formatted at 2026-08-10 19:31:22 on dist-test-slave-2j7r"
server_key: "4bff8a7f67c54ac9a1e51563140a354d"
server_key_iv: "3da66f6d6e5188eb63e61fc771929a1b"
server_key_version: "encryptionkey@0"
I20260810 19:31:22.645035 25596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:22.665223 25596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:22.669166 25596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:22.669456 25596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:22.686782 25596 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:36919
I20260810 19:31:22.686826 25699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:36919 every 8 connection(s)
I20260810 19:31:22.688077 25596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:22.691798 25700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:22.696300 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25596
I20260810 19:31:22.696492 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:22.697026 19728 external_mini_cluster.cc:1428] Setting key 61d5a0554def60e38bcf3f493e201f67
I20260810 19:31:22.703128 25700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0: Bootstrap starting.
I20260810 19:31:22.706558 25700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:22.707912 25700 log.cc:824] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:22.710351 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:22.698815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37379 (local address 127.19.68.62:36919)
0810 19:31:22.699556 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:22.699567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:22.699607 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:31:22.701712 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:22.701735 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:22.702097 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:22.702386 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:22.702958 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:22.704243 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:22.706058 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:22.706750 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:22.706879 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:22.708150 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:22.708158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:22.708171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:22.708754 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:22.708830 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:22.708854 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:22.709028 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:22.709303 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 19:31:22.709455 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":339,"threads_started":1}
I20260810 19:31:22.710807 25700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0: No bootstrap required, opened a new log
I20260810 19:31:22.714059 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:22.718808 25700 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } }
I20260810 19:31:22.719288 25700 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:22.719400 25700 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bb1b83a8fc45e58139794004c727e0, State: Initialized, Role: FOLLOWER
I20260810 19:31:22.719925 25700 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } }
I20260810 19:31:22.720161 25700 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:22.720301 25700 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:22.720474 25700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:22.722766 25700 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } }
I20260810 19:31:22.723255 25700 leader_election.cc:302] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4bb1b83a8fc45e58139794004c727e0; no voters:
I20260810 19:31:22.723924 25700 leader_election.cc:288] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:22.724023 25712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:22.724645 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:22.718254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48935 (local address 127.19.68.62:36919)
0810 19:31:22.718398 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:22.718403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:22.718960 (+ 557us) server_negotiation.cc:407] Connection header received
0810 19:31:22.719121 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:22.719124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:22.719183 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:22.719267 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:22.719851 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:22.720742 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:22.722116 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:22.722555 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:22.722663 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:22.723815 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:22.723822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:22.723826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:22.723842 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:22.723907 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:22.723926 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:22.724029 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:22.724280 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 19:31:22.724332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:31:22.729882 25712 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [term 1 LEADER]: Becoming Leader. State: Replica: a4bb1b83a8fc45e58139794004c727e0, State: Running, Role: LEADER
I20260810 19:31:22.730537 25712 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } }
I20260810 19:31:22.731076 25700 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:22.733812 25720 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4bb1b83a8fc45e58139794004c727e0. Latest consensus state: current_term: 1 leader_uuid: "a4bb1b83a8fc45e58139794004c727e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } } }
I20260810 19:31:22.734239 25720 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:22.734746 25718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4bb1b83a8fc45e58139794004c727e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bb1b83a8fc45e58139794004c727e0" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 36919 } } }
I20260810 19:31:22.735064 25733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:22.735064 25718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:22.740278 25733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:22.749012 25733 catalog_manager.cc:1347] Generated new cluster ID: 596a81d92b114057af71c69e48a01f10
I20260810 19:31:22.749131 25733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:22.762485 25733 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:22.764199 25733 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:22.780949 25733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4bb1b83a8fc45e58139794004c727e0: Generated new TSK 0
I20260810 19:31:22.782065 25733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:22.788668 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25596
W20260810 19:31:22.790052 25695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37379
W20260810 19:31:22.790807 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25596: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:22.790949 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25596
W20260810 19:31:22.819747 25569 connection.cc:535] client connection to 127.19.68.62:36919 recv error: Network error: failed to read from TLS socket (remote: 127.19.68.62:36919): Connection reset by peer (error 104)
2026-08-10T19:31:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2181 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:22 dist-test-slave-2j7r krb5kdc[25749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:22 dist-test-slave-2j7r krb5kdc[25749](info): set up 2 sockets
Aug 10 19:31:22 dist-test-slave-2j7r krb5kdc[25749](info): commencing operation
krb5kdc: starting...
W20260810 19:31:24.058219 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:24 dist-test-slave-2j7r krb5kdc[25749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:24.381374 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34077
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:24.717052 25775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:24.717381 25775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:24.717499 25775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:24.724730 25775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:24.724846 25775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:24.724906 25775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:24.724975 25775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:24.725026 25775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:24.735005 25775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:34077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:24.736258 25775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:24.737646 25775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:24.749480 25787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:24.749524 25785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:24.749395 25790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:24.750245 25775 server_base.cc:1034] running on GCE node
I20260810 19:31:24.751034 25775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:24.752452 25775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:24.753700 25775 hybrid_clock.cc:648] HybridClock initialized: now 1786390284753648 us; error 39 us; skew 500 ppm
I20260810 19:31:24.754055 25775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:24 dist-test-slave-2j7r krb5kdc[25749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390284, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:24.757077 25775 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:24.758549 25775 webserver.cc:460] Webserver started at http://127.19.68.62:43649/ using document root <none> and password file <none>
I20260810 19:31:24.759193 25775 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:24.759311 25775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:24.759606 25775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:24.762557 25775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "1517d1608a8546c1836006b89f2acd26"
format_stamp: "Formatted at 2026-08-10 19:31:24 on dist-test-slave-2j7r"
server_key: "f05cd98bc2b22f78068862c8fb704cbf"
server_key_iv: "95365dae767c219286d53d4e419b1415"
server_key_version: "encryptionkey@0"
I20260810 19:31:24.763285 25775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "1517d1608a8546c1836006b89f2acd26"
format_stamp: "Formatted at 2026-08-10 19:31:24 on dist-test-slave-2j7r"
server_key: "f05cd98bc2b22f78068862c8fb704cbf"
server_key_iv: "95365dae767c219286d53d4e419b1415"
server_key_version: "encryptionkey@0"
I20260810 19:31:24.768149 25775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:31:24.771359 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:24.772567 25775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:31:24.772722 25775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "1517d1608a8546c1836006b89f2acd26"
format_stamp: "Formatted at 2026-08-10 19:31:24 on dist-test-slave-2j7r"
server_key: "f05cd98bc2b22f78068862c8fb704cbf"
server_key_iv: "95365dae767c219286d53d4e419b1415"
server_key_version: "encryptionkey@0"
I20260810 19:31:24.772889 25775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:24.778367 25775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:24.782166 25775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:24.782420 25775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:24.799816 25775 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:34077
I20260810 19:31:24.799916 25880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:34077 every 8 connection(s)
I20260810 19:31:24.801182 25775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:24.804661 25881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:24.810720 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25775
I20260810 19:31:24.810868 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:24.811336 19728 external_mini_cluster.cc:1428] Setting key da76f3a1e89805522ca248e2d15a6695
I20260810 19:31:24.815555 25881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26: Bootstrap starting.
I20260810 19:31:24.819375 25881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:24.819917 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:24.813110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55773 (local address 127.19.68.62:34077)
0810 19:31:24.813941 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:24.813956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:24.814008 (+ 52us) server_negotiation.cc:407] Connection header received
0810 19:31:24.816262 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:24.816289 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:24.816675 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:24.817132 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:24.818100 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:24.818107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:24.818123 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:24.818140 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:24.818208 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:24.818234 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:24.818448 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:24.818569 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:31:24.818743 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":324,"threads_started":1}
I20260810 19:31:24.820598 25881 log.cc:824] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:24.823350 25881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26: No bootstrap required, opened a new log
I20260810 19:31:24.823378 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:24.829576 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:24.827456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54467 (local address 127.19.68.62:34077)
0810 19:31:24.827625 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:24.827631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:24.827932 (+ 301us) server_negotiation.cc:407] Connection header received
0810 19:31:24.828043 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:24.828048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:24.828105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:24.828206 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:24.828966 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:24.828975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:24.828979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:24.828998 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:24.829046 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:24.829066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:24.829212 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:24.829258 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 19:31:24.829314 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:31:24.831290 25881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } }
I20260810 19:31:24.831624 25881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:24.831696 25881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1517d1608a8546c1836006b89f2acd26, State: Initialized, Role: FOLLOWER
I20260810 19:31:24.832198 25881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } }
I20260810 19:31:24.832465 25881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:24.832610 25881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:24.832787 25881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:24.834923 25881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } }
I20260810 19:31:24.835367 25881 leader_election.cc:302] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1517d1608a8546c1836006b89f2acd26; no voters:
I20260810 19:31:24.835871 25881 leader_election.cc:288] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:24.836004 25896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:24.837409 25896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [term 1 LEADER]: Becoming Leader. State: Replica: 1517d1608a8546c1836006b89f2acd26, State: Running, Role: LEADER
I20260810 19:31:24.837873 25881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:24.837908 25896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } }
I20260810 19:31:24.840715 25897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1517d1608a8546c1836006b89f2acd26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } } }
I20260810 19:31:24.840687 25898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1517d1608a8546c1836006b89f2acd26. Latest consensus state: current_term: 1 leader_uuid: "1517d1608a8546c1836006b89f2acd26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1517d1608a8546c1836006b89f2acd26" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 34077 } } }
I20260810 19:31:24.841176 25898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:24.841174 25897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:24.842329 25904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:24.847505 25904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:24.856256 25904 catalog_manager.cc:1347] Generated new cluster ID: 82830dd4678f4ff3beb2ef623395eb27
I20260810 19:31:24.856383 25904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:24.865968 25904 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:24.866449 25904 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:24.878168 25904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1517d1608a8546c1836006b89f2acd26: Generated new TSK 0
I20260810 19:31:24.878921 25904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:24.893929 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25775
W20260810 19:31:24.894887 25875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55773
W20260810 19:31:24.895550 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:24.895679 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25775
2026-08-10T19:31:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:25 dist-test-slave-2j7r krb5kdc[25932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:25 dist-test-slave-2j7r krb5kdc[25932](info): set up 2 sockets
Aug 10 19:31:25 dist-test-slave-2j7r krb5kdc[25932](info): commencing operation
krb5kdc: starting...
W20260810 19:31:26.172314 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:26 dist-test-slave-2j7r krb5kdc[25932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:26.506057 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:46339
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:26.839057 25952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:26.839350 25952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:26.839491 25952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:26.846585 25952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:26.846702 25952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:26.846781 25952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:26.846865 25952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:26.846925 25952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:26.856438 25952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:26.857713 25952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:26.859030 25952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:26.871668 25963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:26.871935 25964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:26.871742 25968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:26.872792 25952 server_base.cc:1034] running on GCE node
I20260810 19:31:26.873590 25952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:26.874913 25952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:26.876118 25952 hybrid_clock.cc:648] HybridClock initialized: now 1786390286876074 us; error 39 us; skew 500 ppm
I20260810 19:31:26.876456 25952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:26 dist-test-slave-2j7r krb5kdc[25932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390286, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:26.885134 25952 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:26.886698 25952 webserver.cc:460] Webserver started at http://127.19.68.62:43795/ using document root <none> and password file <none>
I20260810 19:31:26.887363 25952 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:26.887485 25952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:26.887776 25952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:26.890786 25952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "8601615f374e44779289d2b4eb69d80d"
format_stamp: "Formatted at 2026-08-10 19:31:26 on dist-test-slave-2j7r"
server_key: "869bdf1789325c913444148faeb5d7ff"
server_key_iv: "098e104efee2bdbc4cd854480ab9f6dd"
server_key_version: "encryptionkey@0"
I20260810 19:31:26.891533 25952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "8601615f374e44779289d2b4eb69d80d"
format_stamp: "Formatted at 2026-08-10 19:31:26 on dist-test-slave-2j7r"
server_key: "869bdf1789325c913444148faeb5d7ff"
server_key_iv: "098e104efee2bdbc4cd854480ab9f6dd"
server_key_version: "encryptionkey@0"
I20260810 19:31:26.896711 25952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:31:26.900061 25978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:26.901371 25952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:26.901521 25952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "8601615f374e44779289d2b4eb69d80d"
format_stamp: "Formatted at 2026-08-10 19:31:26 on dist-test-slave-2j7r"
server_key: "869bdf1789325c913444148faeb5d7ff"
server_key_iv: "098e104efee2bdbc4cd854480ab9f6dd"
server_key_version: "encryptionkey@0"
I20260810 19:31:26.901691 25952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:26.918649 25952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:26.922842 25952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:26.923125 25952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:26.940831 25952 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:46339
I20260810 19:31:26.940864 26061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:46339 every 8 connection(s)
I20260810 19:31:26.942255 25952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:26.945938 26064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:26.947619 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 25952
I20260810 19:31:26.947816 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:26.948379 19728 external_mini_cluster.cc:1428] Setting key acb1f53da31876bb1e6e3ea5849ffdd5
I20260810 19:31:26.960178 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d: Bootstrap starting.
Aug 10 19:31:26 dist-test-slave-2j7r krb5kdc[25932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:26.963795 26064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:26.964828 26064 log.cc:824] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:26.967633 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d: No bootstrap required, opened a new log
I20260810 19:31:26.971112 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:26.950229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57585 (local address 127.19.68.62:46339)
0810 19:31:26.951046 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:26.951060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:26.951101 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:31:26.952721 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:26.952746 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:26.953190 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:26.953533 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:26.955297 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:26.956596 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:26.958458 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:26.959103 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:26.959237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:26.964081 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:26.964088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:26.964101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:26.964141 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:26.967098 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:26.967773 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:26.967782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:26.967790 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:26.967965 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:26.968335 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:26.968338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:26.968340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:26.968863 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:26.969206 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:26.969701 (+ 495us) server_negotiation.cc:299] Negotiation successful
0810 19:31:26.970104 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":375,"threads_started":1}
I20260810 19:31:26.974681 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:26.975739 26064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } }
I20260810 19:31:26.976074 26064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:26.976158 26064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8601615f374e44779289d2b4eb69d80d, State: Initialized, Role: FOLLOWER
I20260810 19:31:26.976698 26064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } }
I20260810 19:31:26.976925 26064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:26.977074 26064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:26.977250 26064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:26.979562 26064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } }
I20260810 19:31:26.980086 26064 leader_election.cc:302] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8601615f374e44779289d2b4eb69d80d; no voters:
I20260810 19:31:26.980664 26064 leader_election.cc:288] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:26.980778 26082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:26.982918 26064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:26.984184 26082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [term 1 LEADER]: Becoming Leader. State: Replica: 8601615f374e44779289d2b4eb69d80d, State: Running, Role: LEADER
I20260810 19:31:26.984877 26082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } }
I20260810 19:31:26.991828 26084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8601615f374e44779289d2b4eb69d80d. Latest consensus state: current_term: 1 leader_uuid: "8601615f374e44779289d2b4eb69d80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } } }
I20260810 19:31:26.992209 26084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:26.993532 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8601615f374e44779289d2b4eb69d80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601615f374e44779289d2b4eb69d80d" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 46339 } } }
I20260810 19:31:26.993873 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:26.994894 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:26.978853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51935 (local address 127.19.68.62:46339)
0810 19:31:26.979023 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:26.979029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:26.979483 (+ 454us) server_negotiation.cc:407] Connection header received
0810 19:31:26.979612 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:26.979617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:26.979692 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:26.979791 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:26.981314 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:26.982189 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:26.983546 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:26.984082 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:26.984222 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:26.987419 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:26.987427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:26.987432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:26.987473 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:26.991247 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:26.992103 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:26.992109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:26.992113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:26.992264 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:26.992801 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:26.992807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:26.992811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:26.993284 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:26.993548 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:26.994232 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 19:31:26.994477 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260810 19:31:26.995246 26092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:27.004304 26092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:27.013044 26092 catalog_manager.cc:1347] Generated new cluster ID: 7ac32158d2c346fa9a984ac3f47c7eb7
I20260810 19:31:27.013206 26092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:27.047098 26092 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:27.048615 26092 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:27.056206 26092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8601615f374e44779289d2b4eb69d80d: Generated new TSK 0
I20260810 19:31:27.057104 26092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:27.072151 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 25952
I20260810 19:31:27.073354 26056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:27.164752 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 25952
2026-08-10T19:31:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:27 dist-test-slave-2j7r krb5kdc[26116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:27 dist-test-slave-2j7r krb5kdc[26116](info): set up 2 sockets
Aug 10 19:31:27 dist-test-slave-2j7r krb5kdc[26116](info): commencing operation
krb5kdc: starting...
W20260810 19:31:28.435501 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.006s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:28 dist-test-slave-2j7r krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:28.749912 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:45933
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:45933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:29.087328 26134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:29.087612 26134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:29.087749 26134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:29.095014 26134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:29.095124 26134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:29.095199 26134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:29.095296 26134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:29.095359 26134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:29.105214 26134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:45933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:45933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:29.106547 26134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:29.107873 26134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:29.119458 26141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:29.119499 26143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:29.119596 26146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:29.120313 26134 server_base.cc:1034] running on GCE node
I20260810 19:31:29.121078 26134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:29.122521 26134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:29.123719 26134 hybrid_clock.cc:648] HybridClock initialized: now 1786390289123677 us; error 31 us; skew 500 ppm
I20260810 19:31:29.124047 26134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:29 dist-test-slave-2j7r krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390289, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:29.127152 26134 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:29.128525 26134 webserver.cc:460] Webserver started at http://127.19.68.62:32837/ using document root <none> and password file <none>
I20260810 19:31:29.129262 26134 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:29.129371 26134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:29.129631 26134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:29.132632 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "53e3e7ea6dc54280b59dab3ec370d7d3"
format_stamp: "Formatted at 2026-08-10 19:31:29 on dist-test-slave-2j7r"
server_key: "065d8a6ad60491560a2086b8bf4596cc"
server_key_iv: "0f12ba88f946f8036dcc9c9240a91f8c"
server_key_version: "encryptionkey@0"
I20260810 19:31:29.133378 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "53e3e7ea6dc54280b59dab3ec370d7d3"
format_stamp: "Formatted at 2026-08-10 19:31:29 on dist-test-slave-2j7r"
server_key: "065d8a6ad60491560a2086b8bf4596cc"
server_key_iv: "0f12ba88f946f8036dcc9c9240a91f8c"
server_key_version: "encryptionkey@0"
I20260810 19:31:29.138388 26134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:31:29.141857 26159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:29.143169 26134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:31:29.143319 26134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "53e3e7ea6dc54280b59dab3ec370d7d3"
format_stamp: "Formatted at 2026-08-10 19:31:29 on dist-test-slave-2j7r"
server_key: "065d8a6ad60491560a2086b8bf4596cc"
server_key_iv: "0f12ba88f946f8036dcc9c9240a91f8c"
server_key_version: "encryptionkey@0"
I20260810 19:31:29.143496 26134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:29.159145 26134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:29.163177 26134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:29.163434 26134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:29.181547 26134 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:45933
I20260810 19:31:29.181608 26240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:45933 every 8 connection(s)
I20260810 19:31:29.182873 26134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:29.186554 26241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:29.190083 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26134
I20260810 19:31:29.190237 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:29.190659 19728 external_mini_cluster.cc:1428] Setting key 2c77a040fc2ebb7c200aac92956fbce6
I20260810 19:31:29.199501 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3: Bootstrap starting.
I20260810 19:31:29.203493 26241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:29.204877 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:29.192329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42119 (local address 127.19.68.62:45933)
0810 19:31:29.193140 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:29.193155 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:29.193205 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:31:29.195074 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:29.195100 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:29.195596 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:29.196024 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:29.196524 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:29.198561 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:29.199862 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:29.200854 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:29.201061 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:29.202129 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:29.202141 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:29.202158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:29.202869 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:29.202964 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:29.202996 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:29.203119 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:29.203270 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 19:31:29.203552 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":310,"threads_started":1}
I20260810 19:31:29.204823 26241 log.cc:824] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:29.208290 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3: No bootstrap required, opened a new log
I20260810 19:31:29.208936 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:29.216332 26241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } }
I20260810 19:31:29.216712 26241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:29.216836 26241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e3e7ea6dc54280b59dab3ec370d7d3, State: Initialized, Role: FOLLOWER
I20260810 19:31:29.217583 26241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } }
I20260810 19:31:29.217883 26241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:29.218024 26241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:29.218182 26241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:29.219456 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:29.213322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42209 (local address 127.19.68.62:45933)
0810 19:31:29.213502 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:29.213507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:29.213884 (+ 377us) server_negotiation.cc:407] Connection header received
0810 19:31:29.214070 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:29.214074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:29.214145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:29.214238 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:29.214764 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:29.215500 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:29.216836 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:29.217519 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:29.217661 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:29.218742 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:29.218748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:29.218753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:29.218781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:29.218834 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:29.218855 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:29.218963 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:29.219036 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 19:31:29.219132 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:31:29.221359 26241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } }
I20260810 19:31:29.221824 26241 leader_election.cc:302] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53e3e7ea6dc54280b59dab3ec370d7d3; no voters:
I20260810 19:31:29.222404 26241 leader_election.cc:288] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:29.222518 26257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:29.223153 26257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [term 1 LEADER]: Becoming Leader. State: Replica: 53e3e7ea6dc54280b59dab3ec370d7d3, State: Running, Role: LEADER
I20260810 19:31:29.223798 26257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } }
I20260810 19:31:29.224743 26241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:29.227041 26258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } } }
I20260810 19:31:29.227002 26259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53e3e7ea6dc54280b59dab3ec370d7d3. Latest consensus state: current_term: 1 leader_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e3e7ea6dc54280b59dab3ec370d7d3" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 45933 } } }
I20260810 19:31:29.227432 26259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:29.227432 26258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:29.229514 26268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:29.233831 26268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:29.242758 26268 catalog_manager.cc:1347] Generated new cluster ID: dfddf6fb01094fa2b5046286d0fb3bed
I20260810 19:31:29.242880 26268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:29.260121 26268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:29.262068 26268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:29.286731 26268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53e3e7ea6dc54280b59dab3ec370d7d3: Generated new TSK 0
I20260810 19:31:29.287624 26268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:29.300400 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26134
W20260810 19:31:29.301882 26235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42119
W20260810 19:31:29.302596 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:29.302740 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26134
2026-08-10T19:31:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:29 dist-test-slave-2j7r krb5kdc[26292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:29 dist-test-slave-2j7r krb5kdc[26292](info): set up 2 sockets
Aug 10 19:31:29 dist-test-slave-2j7r krb5kdc[26292](info): commencing operation
krb5kdc: starting...
W20260810 19:31:30.567844 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.005s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:30 dist-test-slave-2j7r krb5kdc[26292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:30.900698 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:41949
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:31.232970 26318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:31.233297 26318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:31.233439 26318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:31.240634 26318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:31.240746 26318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:31.240834 26318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:31.240916 26318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:31.240975 26318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:31.250537 26318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:31.251771 26318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:31.253062 26318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:31.264371 26327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:31.264302 26328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:31.264276 26331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:31.265342 26318 server_base.cc:1034] running on GCE node
I20260810 19:31:31.266140 26318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:31.267470 26318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:31.268662 26318 hybrid_clock.cc:648] HybridClock initialized: now 1786390291268622 us; error 32 us; skew 500 ppm
I20260810 19:31:31.269001 26318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:31 dist-test-slave-2j7r krb5kdc[26292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390291, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:31.272092 26318 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:31.273557 26318 webserver.cc:460] Webserver started at http://127.19.68.62:39871/ using document root <none> and password file <none>
I20260810 19:31:31.274200 26318 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:31.274312 26318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:31.274600 26318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:31.277942 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "f17eae7dd2ab4951b07fa9109ac55426"
format_stamp: "Formatted at 2026-08-10 19:31:31 on dist-test-slave-2j7r"
server_key: "c1ebf9dab033a633864e77884c374c09"
server_key_iv: "148cfc9106886828fc72d60ad3f6248b"
server_key_version: "encryptionkey@0"
I20260810 19:31:31.278671 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "f17eae7dd2ab4951b07fa9109ac55426"
format_stamp: "Formatted at 2026-08-10 19:31:31 on dist-test-slave-2j7r"
server_key: "c1ebf9dab033a633864e77884c374c09"
server_key_iv: "148cfc9106886828fc72d60ad3f6248b"
server_key_version: "encryptionkey@0"
I20260810 19:31:31.283715 26318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:31:31.287134 26345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:31.288448 26318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:31:31.288631 26318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "f17eae7dd2ab4951b07fa9109ac55426"
format_stamp: "Formatted at 2026-08-10 19:31:31 on dist-test-slave-2j7r"
server_key: "c1ebf9dab033a633864e77884c374c09"
server_key_iv: "148cfc9106886828fc72d60ad3f6248b"
server_key_version: "encryptionkey@0"
I20260810 19:31:31.288842 26318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:31.294721 26318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:31.298645 26318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:31.298946 26318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:31.317734 26318 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:41949
I20260810 19:31:31.317765 26434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:41949 every 8 connection(s)
I20260810 19:31:31.319118 26318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:31.321157 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26318
I20260810 19:31:31.321349 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:31.321907 19728 external_mini_cluster.cc:1428] Setting key ebc1d3f09a198c19ac645da2661d6623
I20260810 19:31:31.323081 26435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:31.329519 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:31.323768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57593 (local address 127.19.68.62:41949)
0810 19:31:31.324549 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:31.324561 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:31.324600 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:31:31.326200 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:31.326221 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:31.326549 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:31.326895 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:31.327921 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:31.327928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:31.327940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:31.327962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:31.328036 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:31.328056 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:31.328266 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:31.328418 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 19:31:31.328583 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":345,"threads_started":1}
I20260810 19:31:31.332846 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:31.334916 26435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426: Bootstrap starting.
I20260810 19:31:31.338364 26435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:31.339316 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:31.337007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48099 (local address 127.19.68.62:41949)
0810 19:31:31.337194 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:31.337201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:31.337580 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:31:31.337748 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:31.337751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:31.337795 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:31.337862 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:31.338755 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:31.338759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:31.338762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:31:31.338779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:31.338818 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:31:31.338834 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:31.338916 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:31.339027 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 19:31:31.339068 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:31:31.339684 26435 log.cc:824] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:31.342585 26435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426: No bootstrap required, opened a new log
I20260810 19:31:31.349745 26435 raft_consensus.cc:348] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } }
I20260810 19:31:31.350111 26435 raft_consensus.cc:374] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:31:31.350214 26435 raft_consensus.cc:729] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17eae7dd2ab4951b07fa9109ac55426, State: Initialized, Role: FOLLOWER
I20260810 19:31:31.350785 26435 consensus_queue.cc:260] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } }
I20260810 19:31:31.351006 26435 raft_consensus.cc:388] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:31:31.351131 26435 raft_consensus.cc:482] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:31:31.351276 26435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:31.353359 26435 raft_consensus.cc:504] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } }
I20260810 19:31:31.353816 26435 leader_election.cc:302] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f17eae7dd2ab4951b07fa9109ac55426; no voters:
I20260810 19:31:31.354377 26435 leader_election.cc:288] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:31:31.354494 26453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:31.355058 26453 raft_consensus.cc:686] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [term 1 LEADER]: Becoming Leader. State: Replica: f17eae7dd2ab4951b07fa9109ac55426, State: Running, Role: LEADER
I20260810 19:31:31.355585 26453 consensus_queue.cc:237] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } }
I20260810 19:31:31.357360 26453 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f17eae7dd2ab4951b07fa9109ac55426. Latest consensus state: current_term: 1 leader_uuid: "f17eae7dd2ab4951b07fa9109ac55426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } } }
I20260810 19:31:31.357621 26453 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:31.358253 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f17eae7dd2ab4951b07fa9109ac55426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17eae7dd2ab4951b07fa9109ac55426" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 41949 } } }
I20260810 19:31:31.358680 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:31.362118 26435 sys_catalog.cc:564] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:31.362330 26459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:31.367216 26459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:31.376314 26459 catalog_manager.cc:1347] Generated new cluster ID: 72e01209c54740d485c4dbda3455c634
I20260810 19:31:31.376430 26459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:31.394462 26459 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:31.394913 26459 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:31.402122 26459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f17eae7dd2ab4951b07fa9109ac55426: Generated new TSK 0
I20260810 19:31:31.403085 26459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:31.411008 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26318
W20260810 19:31:31.412104 26429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57593
W20260810 19:31:31.412751 19728 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:31:31.412906 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26318
2026-08-10T19:31:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2103 ms)
[----------] 18 tests from AuthTokenIssuingTest (39109 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:31 dist-test-slave-2j7r krb5kdc[26486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:31 dist-test-slave-2j7r krb5kdc[26486](info): set up 2 sockets
Aug 10 19:31:31 dist-test-slave-2j7r krb5kdc[26486](info): commencing operation
krb5kdc: starting...
W20260810 19:31:32.687296 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:32 dist-test-slave-2j7r krb5kdc[26486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:33.021368 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35915
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:33.355530 26510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:33.355849 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:33.355949 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:33.356061 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:33.363032 26510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:33.363121 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:33.363200 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:33.363265 26510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:33.363348 26510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:33.372839 26510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:33.374099 26510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:33.375432 26510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:33.386343 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:33.386487 26521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:33.386533 26525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:33.387480 26510 server_base.cc:1034] running on GCE node
I20260810 19:31:33.388162 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:33.389436 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:33.390653 26510 hybrid_clock.cc:648] HybridClock initialized: now 1786390293390595 us; error 33 us; skew 500 ppm
I20260810 19:31:33.390959 26510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:33 dist-test-slave-2j7r krb5kdc[26486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:33.396736 26510 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:33.398149 26510 webserver.cc:460] Webserver started at http://127.19.68.62:33721/ using document root <none> and password file <none>
I20260810 19:31:33.398805 26510 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:33.398909 26510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:33.399206 26510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:33.402179 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "2b7f56eb020d4b5088da9053ddd54c52"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d1cc9332d79976398c7baff2215bb0ef"
server_key_iv: "872f9ff7592d835a2423be620e3069df"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.402982 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "2b7f56eb020d4b5088da9053ddd54c52"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d1cc9332d79976398c7baff2215bb0ef"
server_key_iv: "872f9ff7592d835a2423be620e3069df"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.407961 26510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:31:33.411260 26536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:33.412495 26510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:31:33.412644 26510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "2b7f56eb020d4b5088da9053ddd54c52"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d1cc9332d79976398c7baff2215bb0ef"
server_key_iv: "872f9ff7592d835a2423be620e3069df"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.412842 26510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:33.453856 26510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:33.457695 26510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:33.457960 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:33.475199 26510 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:35915
I20260810 19:31:33.475221 26626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:35915 every 8 connection(s)
I20260810 19:31:33.476496 26510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:33.480146 26627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:33.483888 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26510
I20260810 19:31:33.484133 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:33.484943 19728 external_mini_cluster.cc:1428] Setting key fbe6b918fdb35c13a65185d80b719ac5
I20260810 19:31:33.487986 26627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:33 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:33.511440 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.492895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.19.68.62:35915)
0810 19:31:33.493710 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:33.493718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:33.493772 (+ 54us) server_negotiation.cc:407] Connection header received
0810 19:31:33.495085 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:33.495092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:33.495362 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:33.495583 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:33.496780 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:33.498413 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:33.499928 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:33.500674 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:33.500842 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:33.504660 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:33.504667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:33.504679 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:33.504711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:33.507291 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:33.507933 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:33.507939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:33.507941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:33.508085 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:33.508451 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:33.508454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:33.508456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:33.508911 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:33.509117 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:33.509621 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 19:31:33.509992 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":606,"thread_start_us":434,"threads_started":1}
I20260810 19:31:33.511458 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.491571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35915 (local address 127.0.0.1:47296)
0810 19:31:33.492431 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.492490 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:33.494703 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:33.495604 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:33.495608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:33.496193 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:33.496562 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:33.496580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:33.498598 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:33.498602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:33.499777 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:33.499790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:33.500905 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:33.500910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:33.501154 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:33.501840 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:33.501873 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:33.504458 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:33.507416 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:33.507427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:33.507438 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:33.507813 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:33.508189 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:33.508193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:33.508197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:33.508346 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:33.509046 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:33.509053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:33.509266 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 19:31:33.509484 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:31:33.510132 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":140,"thread_start_us":368,"threads_started":1}
I20260810 19:31:33.514904 26627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:33.516088 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.515646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53156)
0810 19:31:33.515871 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.515980 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 19:31:33.516386 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
I20260810 19:31:33.518038 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.517679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53160)
0810 19:31:33.517820 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.517886 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:31:33.518250 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:38619: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
I20260810 19:31:33.560710 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:38619
--webserver_interface=127.19.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:31:33.567631 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } attempt: 1
I20260810 19:31:33.568910 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.568316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53170)
0810 19:31:33.568580 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.568700 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 19:31:33.570034 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.569603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53176)
0810 19:31:33.569805 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.569892 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:31:33.570261 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:38619: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
I20260810 19:31:33.638509 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } attempt: 2
I20260810 19:31:33.639629 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.639209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53192)
0810 19:31:33.639395 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.639486 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:31:33.640648 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.640289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53196)
0810 19:31:33.640432 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.640516 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:31:33.640825 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:38619: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
I20260810 19:31:33.732076 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } attempt: 3
I20260810 19:31:33.733294 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.732820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53204)
0810 19:31:33.733017 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.733144 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:31:33.734242 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.733894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53210)
0810 19:31:33.734033 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.734103 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:31:33.734421 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:38619: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
I20260810 19:31:33.877652 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } attempt: 4
I20260810 19:31:33.878872 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.878332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53222)
0810 19:31:33.878651 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.878738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 19:31:33.879858 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:33.879498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53226)
0810 19:31:33.879643 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:33.879728 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:31:33.880059 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:38619: Network error: Client connection negotiation failed: client connection to 127.19.68.61:38619: connect: Connection refused (error 111)
W20260810 19:31:33.890915 26639 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:33.891258 26639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:33.891358 26639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:33.891469 26639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:33.898665 26639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:33.898757 26639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:33.898845 26639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:33.898909 26639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:33.898993 26639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:33.908728 26639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:33.910012 26639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:33.911355 26639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:33.922705 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:33.922716 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:33.922770 26654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:33.923861 26639 server_base.cc:1034] running on GCE node
I20260810 19:31:33.924626 26639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:33.926029 26639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:33.927274 26639 hybrid_clock.cc:648] HybridClock initialized: now 1786390293927177 us; error 45 us; skew 500 ppm
I20260810 19:31:33.927582 26639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:33 dist-test-slave-2j7r krb5kdc[26486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:33.930634 26639 init.cc:377] Logged in from keytab as kudu/127.19.68.61@KRBTEST.COM (short username kudu)
I20260810 19:31:33.932041 26639 webserver.cc:460] Webserver started at http://127.19.68.61:46387/ using document root <none> and password file <none>
I20260810 19:31:33.932709 26639 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:33.932798 26639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:33.933063 26639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:33.935974 26639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data/instance:
uuid: "c3e18f266ca042c897d97bb365e6ab03"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d7022ab6a8e1a9be70a8bc911a2bac29"
server_key_iv: "df255fc7513db86e87ad986748e20299"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.936697 26639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal/instance:
uuid: "c3e18f266ca042c897d97bb365e6ab03"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d7022ab6a8e1a9be70a8bc911a2bac29"
server_key_iv: "df255fc7513db86e87ad986748e20299"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.941753 26639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:31:33.945309 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:33.946805 26639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:31:33.946970 26639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "c3e18f266ca042c897d97bb365e6ab03"
format_stamp: "Formatted at 2026-08-10 19:31:33 on dist-test-slave-2j7r"
server_key: "d7022ab6a8e1a9be70a8bc911a2bac29"
server_key_iv: "df255fc7513db86e87ad986748e20299"
server_key_version: "encryptionkey@0"
I20260810 19:31:33.947180 26639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:33.969642 26639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:33.973831 26639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:33.974080 26639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:33.991777 26639 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.61:38619
I20260810 19:31:33.991797 26752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.61:38619 every 8 connection(s)
I20260810 19:31:33.993048 26639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
I20260810 19:31:33.993441 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26639
I20260810 19:31:33.993602 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-1/wal/instance
I20260810 19:31:33.994141 19728 external_mini_cluster.cc:1428] Setting key fd28009c82cb83945a8296bb30018603
I20260810 19:31:33.996703 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:34.003100 26754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:34.023741 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.005710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.19.68.62:35915)
0810 19:31:34.005916 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.005922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.007075 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 19:31:34.009393 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.009399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.009479 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.009566 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.011009 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.011692 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.013211 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.013625 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.013749 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.017520 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.017525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.017528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.017562 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.020152 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.021051 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.021056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.021058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.021228 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.021755 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.021760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.021762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.022093 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.022248 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.023273 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.023511 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:31:34.025040 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.006093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35915 (local address 127.0.0.1:47298)
0810 19:31:34.006888 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.006946 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.009058 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.009800 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.009808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.010314 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.010841 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.010858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.011852 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.011856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.013049 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.013061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.013849 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.013856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.014069 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.014676 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.014704 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.017292 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.020307 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.020316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.020332 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.020897 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.021352 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.021356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.021360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.021587 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.022227 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.022235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.022740 (+ 505us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.023061 (+ 321us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.023583 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
WARNING: no policy specified for kudu/127.19.68.60@KRBTEST.COM; defaulting to no policy
I20260810 19:31:34.026949 26754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.60@KRBTEST.COM" created.
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:34.040592 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.027738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53230)
0810 19:31:34.027925 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.027941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.028023 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.028968 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.028972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.029391 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.029522 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.029531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.031016 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.031020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.031810 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.031831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.032639 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.032645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.032801 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.033350 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.033366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.035375 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.038318 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.038331 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.038335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.038675 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.039020 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.039025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.039029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.039157 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.039703 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.039708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.039857 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.039969 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.040212 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 19:31:34.040690 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.027970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.19.68.61:38619)
0810 19:31:34.028586 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.028591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.028616 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:31:34.028656 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.028661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.028823 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.028953 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.029682 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.030874 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.031947 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.032475 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.032596 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.035561 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.035572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.035584 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.035616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.038203 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.038792 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.038798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.038800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.038919 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.039253 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.039257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.039259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.039588 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.039768 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.040102 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.040315 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":479,"thread_start_us":323,"threads_started":1}
I20260810 19:31:34.043033 26754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:34.044119 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.043615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60866)
0810 19:31:34.043801 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.043937 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 19:31:34.044369 26671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.045928 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.045606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60876)
0810 19:31:34.045748 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.045824 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:31:34.046139 26754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
I20260810 19:31:34.066479 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:44597
--webserver_interface=127.19.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:31:34.095429 26754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 1
I20260810 19:31:34.096552 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.096089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60886)
0810 19:31:34.096285 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.096395 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 19:31:34.097697 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.097340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60902)
0810 19:31:34.097489 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.097581 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:31:34.097918 26754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.166138 26754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 2
I20260810 19:31:34.167309 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.166846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60914)
0810 19:31:34.167049 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.167149 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 19:31:34.168354 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.168007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60920)
0810 19:31:34.168144 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.168214 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:31:34.168578 26754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.179292 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } attempt: 5
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:34.193068 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.179986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.19.68.61:38619)
0810 19:31:34.180162 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.180166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.180219 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:31:34.180426 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.180430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.180491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.180569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.181562 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.182261 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.184010 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.184429 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.184519 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.188131 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.188136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.188140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.188171 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.190708 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.191447 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.191451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.191453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.191569 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.191914 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.191916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.191918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.192175 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.192282 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.192701 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.192862 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:31:34.193230 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.179924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53240)
0810 19:31:34.180122 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.180146 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.180301 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.180638 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.180643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.181172 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.181433 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.181447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.182431 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.182436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.183857 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.183875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.184578 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.184584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.184814 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.185433 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.185456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.187948 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.190846 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.190851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.190856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.191340 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.191661 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.191665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.191669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.191828 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.192279 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.192284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.192462 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.192626 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.192885 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 19:31:34.194268 26627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:34.195179 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.194836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60926)
0810 19:31:34.194973 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.195045 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:31:34.196017 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.195731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60942)
0810 19:31:34.195852 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.195927 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 19:31:34.196201 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.247404 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 1
I20260810 19:31:34.248621 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.248127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60958)
0810 19:31:34.248347 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.248442 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:31:34.249584 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.249268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60968)
0810 19:31:34.249409 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.249477 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 19:31:34.249794 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.259785 26754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 3
I20260810 19:31:34.260818 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.260393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60984)
0810 19:31:34.260585 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.260670 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 19:31:34.261863 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.261432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:60996)
0810 19:31:34.261660 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.261739 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260810 19:31:34.262068 26754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
I20260810 19:31:34.317987 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 2
I20260810 19:31:34.319123 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.318644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32772)
0810 19:31:34.318862 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.318952 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 19:31:34.320080 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.319777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32784)
0810 19:31:34.319908 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.319974 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 19:31:34.320289 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
W20260810 19:31:34.403633 26762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:34.403970 26762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:34.404064 26762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:34.404174 26762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 19:31:34.405257 26754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 4
I20260810 19:31:34.406451 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.405969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32786)
0810 19:31:34.406188 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.406278 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:31:34.407346 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.407022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32794)
0810 19:31:34.407165 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.407232 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:31:34.407533 26754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
W20260810 19:31:34.411571 26762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:34.411655 26762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:34.411746 26762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:34.411809 26762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:34.411899 26762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:34.421602 26762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35915,127.19.68.61:38619,127.19.68.60:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:34.422811 26762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:34.424052 26762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:31:34.426482 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 3
I20260810 19:31:34.427454 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.427032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32810)
0810 19:31:34.427223 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.427303 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:31:34.428439 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.428061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32814)
0810 19:31:34.428253 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.428331 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260810 19:31:34.428625 26627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:44597: Network error: Client connection negotiation failed: client connection to 127.19.68.60:44597: connect: Connection refused (error 111)
W20260810 19:31:34.435321 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:34.435246 26775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:34.435295 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:34.436110 26762 server_base.cc:1034] running on GCE node
I20260810 19:31:34.436810 26762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:34.438140 26762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:34.439376 26762 hybrid_clock.cc:648] HybridClock initialized: now 1786390294439290 us; error 46 us; skew 500 ppm
I20260810 19:31:34.439668 26762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390294, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:34.442664 26762 init.cc:377] Logged in from keytab as kudu/127.19.68.60@KRBTEST.COM (short username kudu)
I20260810 19:31:34.444000 26762 webserver.cc:460] Webserver started at http://127.19.68.60:42895/ using document root <none> and password file <none>
I20260810 19:31:34.444737 26762 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:34.444839 26762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:34.445182 26762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:34.448045 26762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data/instance:
uuid: "558c1dac57a64165aea082e714bfae23"
format_stamp: "Formatted at 2026-08-10 19:31:34 on dist-test-slave-2j7r"
server_key: "1eb1711ad19f611b5eaeed68ded5e121"
server_key_iv: "d1f5c13cbf0253485f6a17094cfc9fac"
server_key_version: "encryptionkey@0"
I20260810 19:31:34.448851 26762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal/instance:
uuid: "558c1dac57a64165aea082e714bfae23"
format_stamp: "Formatted at 2026-08-10 19:31:34 on dist-test-slave-2j7r"
server_key: "1eb1711ad19f611b5eaeed68ded5e121"
server_key_iv: "d1f5c13cbf0253485f6a17094cfc9fac"
server_key_version: "encryptionkey@0"
I20260810 19:31:34.453825 26762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 19:31:34.457001 26786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:34.458242 26762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:31:34.458423 26762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "558c1dac57a64165aea082e714bfae23"
format_stamp: "Formatted at 2026-08-10 19:31:34 on dist-test-slave-2j7r"
server_key: "1eb1711ad19f611b5eaeed68ded5e121"
server_key_iv: "d1f5c13cbf0253485f6a17094cfc9fac"
server_key_version: "encryptionkey@0"
I20260810 19:31:34.458630 26762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:34.481251 26762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:34.486698 26762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:34.486951 26762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:34.504035 26762 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.60:44597
I20260810 19:31:34.504053 26869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.60:44597 every 8 connection(s)
I20260810 19:31:34.505402 26762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
I20260810 19:31:34.509104 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:34.509577 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 26762
I20260810 19:31:34.509776 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389929706493-19728-0/minicluster-data/master-2/wal/instance
I20260810 19:31:34.510313 19728 external_mini_cluster.cc:1428] Setting key 349b5b30fbb54b317484c742f4ffcb0b
I20260810 19:31:34.518214 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390294, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:34.533787 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.512197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.19.68.60:44597)
0810 19:31:34.512976 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.512990 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.513031 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:31:34.515152 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.515174 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.515532 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.515876 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.517694 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.519148 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.520696 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.521714 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.521871 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.526417 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.526424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.526434 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.526467 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.529745 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.530477 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.530485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.530493 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.530676 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.530998 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.531002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.531004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.531471 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.531757 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.532193 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.532555 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":365,"threads_started":1}
I20260810 19:31:34.535748 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.520796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35915 (local address 127.0.0.1:47314)
0810 19:31:34.521621 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.521664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.521839 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.522247 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.522253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.522954 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.523196 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.523204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.524540 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.524544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.525854 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.525867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.526711 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.526718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.526946 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.527469 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.527496 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.530032 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.533009 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.533018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.533025 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.533575 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.533940 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.533945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.533949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.534134 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.534645 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.534652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.534842 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.535032 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.535321 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":423,"threads_started":1}
I20260810 19:31:34.535761 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.520771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.19.68.62:35915)
0810 19:31:34.520994 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.521001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.521776 (+ 775us) server_negotiation.cc:407] Connection header received
0810 19:31:34.521973 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.521978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.522059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.522153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.523358 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.524402 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.526005 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.526539 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.526629 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.530202 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.530207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.530210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.530244 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.532893 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.533683 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.533687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.533689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.533830 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.534258 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.534262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.534264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.534518 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.534626 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.535173 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.535416 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:31:34.537061 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390294, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:34.550200 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.537826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:38619 (local address 127.0.0.1:53252)
0810 19:31:34.537969 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.537986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.538063 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.538333 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.538336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.538744 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.538917 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.538929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.540029 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.540033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.541011 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.541024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.541877 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.541883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.542097 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.542564 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.542580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.544535 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.547563 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.547568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.547572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.548051 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.548469 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.548474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.548478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.548651 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.549281 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.549286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.549472 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.549587 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.549864 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:31:34.550233 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.537871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.19.68.61:38619)
0810 19:31:34.538062 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.538067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.538088 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:31:34.538143 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.538147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.538221 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.538347 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.539054 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.539902 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.541170 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.541723 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.541850 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.544721 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.544728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.544732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.544773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.547421 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.548184 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.548189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.548191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.548356 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.548760 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.548765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.548768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.549174 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.549297 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.549716 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.549941 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:31:34.551260 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.536553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.19.68.62:35915)
0810 19:31:34.536701 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.536706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.536749 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:31:34.536898 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.536903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.536966 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.537037 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.538429 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.539362 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.540729 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.541306 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.541396 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.545619 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.545626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.545630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.545665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.548404 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.549174 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.549178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.549180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.549325 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.549717 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.549722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.549725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.550084 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.550351 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.550750 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.550999 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:31:34.551674 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390294, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:34.564267 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.552484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.19.68.60:44597)
0810 19:31:34.552620 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.552625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.552642 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:31:34.552826 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.552831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.552893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.552984 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.553838 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.554511 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.555684 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.556070 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.556163 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.559365 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.559369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.559372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.559399 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.561792 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.562451 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.562455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.562457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.562567 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.563002 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.563007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.563011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.563382 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.563487 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.563877 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.564030 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:31:34.564589 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.552426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32832)
0810 19:31:34.552584 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.552601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.552699 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.553019 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.553024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.553512 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.553691 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.553702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.554636 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.554642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.555542 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.555555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.556237 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.556244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.556466 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.557044 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.557065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.559190 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.561912 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.561916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.561920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.562342 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.562687 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.562692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.562696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.562897 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.563503 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.563509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.563664 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.563815 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.564216 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":86}
I20260810 19:31:34.567241 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.552864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.19.68.61:38619)
0810 19:31:34.553030 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.553036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.553115 (+ 79us) server_negotiation.cc:407] Connection header received
0810 19:31:34.553270 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.553273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.553326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.553393 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.554689 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.555370 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.556766 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.557329 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.557421 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.561481 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.561485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.561488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.561517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.564583 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.565374 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.565378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.565380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.565499 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.565941 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.565956 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.565960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.566217 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.566468 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.566736 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.566945 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:31:34.569906 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: Bootstrap starting.
I20260810 19:31:34.571914 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:34.573315 26870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:34.574338 26870 log.cc:824] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:34.576819 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: No bootstrap required, opened a new log
I20260810 19:31:34.574396 26580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2b7f56eb020d4b5088da9053ddd54c52"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47316
I20260810 19:31:34.583850 26870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.584151 26870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558c1dac57a64165aea082e714bfae23, State: Initialized, Role: FOLLOWER
I20260810 19:31:34.584548 26870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.585357 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.585681 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.586584 26870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:31:34.595821 26903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:31:34.595942 26903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:31:34.593883 26762 master_runner.cc:186] Error getting master registration for 127.19.68.62:35915: OK, instance_id { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" instance_seqno: 1786390293470447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:31:34.604647 26762 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:31:34.605844 26627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 4
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
I20260810 19:31:34.620815 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.606585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.19.68.60:44597)
0810 19:31:34.606760 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.606764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.606786 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:31:34.607021 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.607024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.607082 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.607171 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.607979 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.608749 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.610029 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.610497 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.610609 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.614064 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.614071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.614076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.614120 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.617880 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.618659 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.618665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.618669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.618823 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.619241 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.619246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.619250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.619626 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.619785 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.620229 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.620448 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:31:34.620815 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.606499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32842)
0810 19:31:34.606715 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.606738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.606876 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.607232 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.607236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.607650 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.607815 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.607824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.608897 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.608903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.609886 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.609899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.610635 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.610641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.610824 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.611374 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.611392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.613872 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.618013 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.618019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.618023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.618540 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.618943 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.618948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.618952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.619133 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.619764 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.619769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.619949 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.620115 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.620435 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 19:31:34.627033 26627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: Bootstrap starting.
I20260810 19:31:34.631220 26627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:34.632580 26627 log.cc:824] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:34.635998 26627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: No bootstrap required, opened a new log
I20260810 19:31:34.643628 26627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.644057 26627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7f56eb020d4b5088da9053ddd54c52, State: Initialized, Role: FOLLOWER
I20260810 19:31:34.644572 26627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.645569 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.645979 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.646854 26627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:34.646900 26580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:31:34.647078 26580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:34.650103 26580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.653477 26580 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [CANDIDATE]: Term 1 election: Requested vote from peers c3e18f266ca042c897d97bb365e6ab03 (127.19.68.61:38619), 558c1dac57a64165aea082e714bfae23 (127.19.68.60:44597)
I20260810 19:31:34.655185 26830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7f56eb020d4b5088da9053ddd54c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "558c1dac57a64165aea082e714bfae23"
I20260810 19:31:34.655747 26830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:34.659384 26830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7f56eb020d4b5088da9053ddd54c52 in term 1.
I20260810 19:31:34.660080 26540 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7f56eb020d4b5088da9053ddd54c52, 558c1dac57a64165aea082e714bfae23; no voters:
I20260810 19:31:34.660605 26908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:34.661309 26924 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:31:34.669436 26908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7f56eb020d4b5088da9053ddd54c52, State: Running, Role: LEADER
I20260810 19:31:34.670075 26908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.674708 26926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7f56eb020d4b5088da9053ddd54c52. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.675103 26926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:34.685688 26929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:34.677980 26510 master_runner.cc:186] Error getting master registration for 127.19.68.61:38619: OK, instance_id { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" instance_seqno: 1786390293986795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:31:34.689458 26510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:31:34.705196 26929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:34.706720 26754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } attempt: 5
W20260810 19:31:34.710707 26543 tablet.cc:2369] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:31:34.726377 26830 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [term 1 FOLLOWER]: Refusing update from remote peer 2b7f56eb020d4b5088da9053ddd54c52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:31:34.729279 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.699479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.19.68.61:38619)
0810 19:31:34.699669 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.699675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.702590 (+ 2915us) server_negotiation.cc:407] Connection header received
0810 19:31:34.703025 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.703033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.703123 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.703198 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.707446 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.708668 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.711766 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.712434 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.712537 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.718410 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.718417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.718421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.718473 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.722201 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.724471 (+ 2270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.724477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.724481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.724650 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.726046 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.726051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.726054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.726423 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.726668 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.728636 (+ 1968us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.728892 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:31:34.730613 26926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 19:31:34 dist-test-slave-2j7r krb5kdc[26486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390293, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
W20260810 19:31:34.738971 26796 tablet.cc:2369] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:31:34.753727 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.701464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.19.68.62:35915)
0810 19:31:34.701624 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.701629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.714107 (+ 12478us) server_negotiation.cc:407] Connection header received
0810 19:31:34.714531 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.714537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.714622 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.714699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.726957 (+ 12258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.727946 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.730778 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.734906 (+ 4128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.735026 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.743667 (+ 8641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.743674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.743678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.743721 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.748810 (+ 5089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.750179 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.750185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.750189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.750527 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.751192 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.751197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.751200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.751586 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.751829 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.752769 (+ 940us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.753011 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:31:34.765507 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.705851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.19.68.60:44597)
0810 19:31:34.706054 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.706064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.725502 (+ 19438us) server_negotiation.cc:407] Connection header received
0810 19:31:34.726961 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.726965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.727040 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.727114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.733025 (+ 5911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.734166 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.742952 (+ 8786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.743700 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.743810 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.755082 (+ 11272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.755089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.755094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.755127 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.760280 (+ 5153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.761530 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.761536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.761540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.761705 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.762468 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.762474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.762477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.762848 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.763156 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.764211 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.764609 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:31:34.766532 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7f56eb020d4b5088da9053ddd54c52. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.767899 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.768988 26942 mvcc.cc:205] Tried to move back new op lower bound from 7317054647165030400 to 7317054646974251008. Current Snapshot: MvccSnapshot[applied={T|T < 7317054647165030400}]
I20260810 19:31:34.770875 26944 mvcc.cc:205] Tried to move back new op lower bound from 7317054647165030400 to 7317054646974251008. Current Snapshot: MvccSnapshot[applied={T|T < 7317054647165030400}]
I20260810 19:31:34.776968 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.707257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:44597 (local address 127.0.0.1:32862)
0810 19:31:34.707608 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:34.707627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:34.707726 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:34.724654 (+ 16928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:34.724659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:34.725208 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:34.725420 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.725432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.728179 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.728185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.729659 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:34.729675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.734665 (+ 4990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.734672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:34.734895 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.735594 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:34.735617 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:34.740540 (+ 4923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:34.770918 (+ 30378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.770924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.770928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.771521 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.772272 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:34.772278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:34.772281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:34.772462 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:34.775930 (+ 3468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:34.775936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:34.776108 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 19:31:34.776275 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 19:31:34.776588 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260810 19:31:34.779112 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:34.707944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.19.68.60:44597)
0810 19:31:34.724329 (+ 16385us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.724335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:34.724359 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:31:34.724418 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:34.724424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:34.724503 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:34.724625 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:34.725626 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.727950 (+ 2324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.733291 (+ 5341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:34.734421 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:34.754608 (+ 20187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:34.760627 (+ 6019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:34.760637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:34.760642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:34.760713 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:34.770678 (+ 9965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.771776 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.771785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.771789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.772057 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:34.772657 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:34.772662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:34.772666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:34.775726 (+ 3060us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:34.777669 (+ 1943us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:34.777904 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 19:31:34.778311 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16232,"thread_start_us":16867,"threads_started":1}
I20260810 19:31:34.780747 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.781077 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:34.782377 26929 catalog_manager.cc:1347] Generated new cluster ID: ae7ec644d92f43e7b4105abd36bf693b
I20260810 19:31:34.782480 26929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:34.786468 26754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: Bootstrap starting.
I20260810 19:31:34.786784 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.787024 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.794269 26754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:34.799286 26754 log.cc:824] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:34.801582 26929 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:34.807811 26929 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:34.811347 26754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: No bootstrap required, opened a new log
I20260810 19:31:34.823060 26754 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.828405 26929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52: Generated new TSK 0
I20260810 19:31:34.829277 26754 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e18f266ca042c897d97bb365e6ab03, State: Initialized, Role: FOLLOWER
I20260810 19:31:34.829916 26754 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } }
I20260810 19:31:34.830307 26929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:34.835121 26754 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:34.835553 26705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:34.838215 26706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7f56eb020d4b5088da9053ddd54c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3e18f266ca042c897d97bb365e6ab03"
I20260810 19:31:34.839315 26705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [term 1 FOLLOWER]: Refusing update from remote peer 2b7f56eb020d4b5088da9053ddd54c52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:31:34.841334 26908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:31:34.843374 26706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7f56eb020d4b5088da9053ddd54c52 in term 1.
I20260810 19:31:34.833935 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.851636 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.874766 26957 mvcc.cc:205] Tried to move back new op lower bound from 7317054647165030400 to 7317054646974251008. Current Snapshot: MvccSnapshot[applied={T|T < 7317054647165030400}]
I20260810 19:31:34.886055 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7f56eb020d4b5088da9053ddd54c52. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.886430 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.886790 26954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:34.887110 26954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:34.912122 26979 catalog_manager.cc:1259] Loaded cluster ID: ae7ec644d92f43e7b4105abd36bf693b
I20260810 19:31:34.912333 26979 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: loading cluster ID for follower catalog manager: success
I20260810 19:31:34.915931 26979 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: acquiring CA information for follower catalog manager: success
I20260810 19:31:34.919050 26979 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3e18f266ca042c897d97bb365e6ab03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 19:31:34.953204 19728 mini_kdc.cc:348] Time spent kdestroy: real 0.108s user 0.003s sys 0.026s
W20260810 19:31:35.006547 26633 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:34.986806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.19.68.62:35915)
0810 19:31:34.987039 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.987049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.004388 (+ 17339us) server_negotiation.cc:407] Connection header received
0810 19:31:35.004885 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.004891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.004977 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.005045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.006279 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47328: BlockingRecv error: recv got EOF from 127.0.0.1:47328 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 19:31:35.008909 26758 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:34.988849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.19.68.61:38619)
0810 19:31:34.989063 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:34.989069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.007352 (+ 18283us) server_negotiation.cc:407] Connection header received
0810 19:31:35.007603 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.007609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.007687 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.007771 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.008680 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53276: BlockingRecv error: recv got EOF from 127.0.0.1:53276 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 19:31:35.012930 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.010613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.19.68.60:44597)
0810 19:31:35.010776 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.010782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.011242 (+ 460us) server_negotiation.cc:407] Connection header received
0810 19:31:35.011481 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.011486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.011592 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.011664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.012705 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32866: BlockingRecv error: recv got EOF from 127.0.0.1:32866 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 19:31:35.070197 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.050632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.19.68.61:38619)
0810 19:31:35.050900 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.050908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.062101 (+ 11193us) server_negotiation.cc:407] Connection header received
0810 19:31:35.062390 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.062396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.062669 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.062856 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.063674 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.064844 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.066949 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.067635 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.067736 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.068522 (+ 786us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.069312 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.069447 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.069718 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.069828 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260810 19:31:35.115863 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.105850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.19.68.60:44597)
0810 19:31:35.106082 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.106089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.113608 (+ 7519us) server_negotiation.cc:407] Connection header received
0810 19:31:35.114189 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.114201 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.114367 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.114453 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.115583 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32872: BlockingRecv error: recv got EOF from 127.0.0.1:32872 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 19:31:35.164403 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.147834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.19.68.61:38619)
0810 19:31:35.148059 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.148065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.157649 (+ 9584us) server_negotiation.cc:407] Connection header received
0810 19:31:35.157871 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.157876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.157969 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.158059 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.158711 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.159869 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.161991 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.162725 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.162829 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.163515 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.163678 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.163741 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.163991 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.164078 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 19:31:35.216089 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.206330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.19.68.60:44597)
0810 19:31:35.206552 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.206558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.214048 (+ 7490us) server_negotiation.cc:407] Connection header received
0810 19:31:35.214597 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.214604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.214702 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.214780 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.215863 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32886: BlockingRecv error: recv got EOF from 127.0.0.1:32886 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:31:35.235960 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.230135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.19.68.61:38619)
0810 19:31:35.230356 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.230363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.230871 (+ 508us) server_negotiation.cc:407] Connection header received
0810 19:31:35.231066 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.231072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.231149 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.231232 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.231957 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.232752 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.234179 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.234627 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.234742 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.235167 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.235293 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.235374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.235548 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.235629 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260810 19:31:35.244002 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.242236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.19.68.60:44597)
0810 19:31:35.242423 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.242429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.242449 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:31:35.242680 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.242686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.242780 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.242870 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.243778 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32896: BlockingRecv error: recv got EOF from 127.0.0.1:32896 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:31:35.257194 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.252085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.19.68.61:38619)
0810 19:31:35.252289 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.252295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.252542 (+ 247us) server_negotiation.cc:407] Connection header received
0810 19:31:35.252728 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.252734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.252813 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.252886 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.253321 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.254168 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.255477 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.255951 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.256061 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.256470 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.256571 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.256653 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.256799 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.256878 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 19:31:35.268579 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.266516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.19.68.60:44597)
0810 19:31:35.266806 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.266813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.267266 (+ 453us) server_negotiation.cc:407] Connection header received
0810 19:31:35.267428 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.267435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.267513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.267597 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.268334 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 19:31:35.285066 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.279796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.19.68.61:38619)
0810 19:31:35.279994 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.280001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.280491 (+ 490us) server_negotiation.cc:407] Connection header received
0810 19:31:35.280671 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.280675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.280740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.280816 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.281268 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.282035 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.283331 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.283780 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.283893 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.284313 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.284416 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.284501 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.284647 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.284724 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:31:35.287774 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7f56eb020d4b5088da9053ddd54c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7f56eb020d4b5088da9053ddd54c52" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35915 } } peers { permanent_uuid: "c3e18f266ca042c897d97bb365e6ab03" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 38619 } } peers { permanent_uuid: "558c1dac57a64165aea082e714bfae23" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 44597 } } }
I20260810 19:31:35.288130 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7f56eb020d4b5088da9053ddd54c52 [sys.catalog]: This master's current role is: LEADER
W20260810 19:31:35.312258 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.310234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.19.68.60:44597)
0810 19:31:35.310450 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.310457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.310933 (+ 476us) server_negotiation.cc:407] Connection header received
0810 19:31:35.311128 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.311135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.311215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.311302 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.312067 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32922: BlockingRecv error: recv got EOF from 127.0.0.1:32922 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:31:35.337347 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.331687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.19.68.61:38619)
0810 19:31:35.331917 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.331923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.332491 (+ 568us) server_negotiation.cc:407] Connection header received
0810 19:31:35.332679 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.332683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.332754 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.332830 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.333317 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.334172 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.335582 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.336025 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.336138 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.336604 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.336739 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.336831 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.336943 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.337036 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 19:31:35.345561 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.344001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.19.68.60:44597)
0810 19:31:35.344171 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.344176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.344194 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:31:35.344349 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.344355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.344446 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.344503 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.345332 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32932: BlockingRecv error: recv got EOF from 127.0.0.1:32932 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:31:35.386399 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.380575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.19.68.61:38619)
0810 19:31:35.380836 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.380844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.381146 (+ 302us) server_negotiation.cc:407] Connection header received
0810 19:31:35.381352 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.381358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.381455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.381587 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.381960 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.383149 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.384387 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.385108 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.385231 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.385634 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.385770 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.385885 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.386024 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.386110 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 19:31:35.393626 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.391923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.19.68.60:44597)
0810 19:31:35.392104 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.392110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.392328 (+ 218us) server_negotiation.cc:407] Connection header received
0810 19:31:35.392472 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.392478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.392584 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.392684 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.393389 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32934: BlockingRecv error: recv got EOF from 127.0.0.1:32934 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:31:35.466496 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.461139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.19.68.61:38619)
0810 19:31:35.461334 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.461341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.461725 (+ 384us) server_negotiation.cc:407] Connection header received
0810 19:31:35.461941 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.461945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.462067 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.462171 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.462621 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.463512 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.464849 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.465295 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.465408 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.465802 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.465919 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.466007 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.466153 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.466235 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 19:31:35.473872 26941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:35.472103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.19.68.60:44597)
0810 19:31:35.472333 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.472340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.472571 (+ 231us) server_negotiation.cc:407] Connection header received
0810 19:31:35.472719 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.472723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.472794 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.472887 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:35.473673 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32946: BlockingRecv error: recv got EOF from 127.0.0.1:32946 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 19:31:35.598039 26903 catalog_manager.cc:1259] Loaded cluster ID: ae7ec644d92f43e7b4105abd36bf693b
I20260810 19:31:35.598152 26903 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: loading cluster ID for follower catalog manager: success
I20260810 19:31:35.600678 26903 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: acquiring CA information for follower catalog manager: success
I20260810 19:31:35.602484 26903 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 558c1dac57a64165aea082e714bfae23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:31:35.611565 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.606483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.19.68.61:38619)
0810 19:31:35.606684 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.606690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.607041 (+ 351us) server_negotiation.cc:407] Connection header received
0810 19:31:35.607239 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.607243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.607310 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.607397 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.607837 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.608546 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.609939 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.610347 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.610466 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.610874 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.610976 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.611051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.611204 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.611285 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:31:35.622486 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.617191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.19.68.60:44597)
0810 19:31:35.617340 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.617344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.617712 (+ 368us) server_negotiation.cc:407] Connection header received
0810 19:31:35.617850 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.617854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.618007 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.618098 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.618462 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.619159 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.620408 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.620812 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.620909 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.621467 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.621996 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.622101 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.622198 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.622262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:31:35.638886 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.629352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.19.68.60:44597)
0810 19:31:35.629555 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.629561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.633336 (+ 3775us) server_negotiation.cc:407] Connection header received
0810 19:31:35.633470 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.633475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.633555 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.633644 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.634071 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.635026 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.637153 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.637608 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.637715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.638195 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.638342 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.638431 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.638543 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.638622 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:31:35.639739 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.628367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.19.68.62:35915)
0810 19:31:35.633357 (+ 4990us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.633372 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.633389 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:31:35.633505 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.633509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.633729 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.633908 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.634482 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.635474 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.637240 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.637920 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.638044 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.638552 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.639024 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.639125 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.639334 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.639418 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4873,"thread_start_us":4777,"threads_started":1}
I20260810 19:31:35.640746 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:35.628457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.19.68.61:38619)
0810 19:31:35.628654 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:35.628660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:35.632997 (+ 4337us) server_negotiation.cc:407] Connection header received
0810 19:31:35.634300 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:35.634305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:35.634405 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:35.634487 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:35.635093 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.636219 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.638310 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:35.638910 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:35.639026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:35.640018 (+ 992us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:35.640159 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:35.640221 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:35.640416 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 19:31:35.640495 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:31:35.644307 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26510
I20260810 19:31:35.645725 26622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:35.757455 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26510
I20260810 19:31:35.785229 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26639
I20260810 19:31:35.786361 26740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:35.891803 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26639
I20260810 19:31:35.919741 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 26762
I20260810 19:31:35.920882 26864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:36.028921 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 26762
2026-08-10T19:31:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4620 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:31:36 dist-test-slave-2j7r krb5kdc[27171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:31:36 dist-test-slave-2j7r krb5kdc[27171](info): set up 2 sockets
Aug 10 19:31:36 dist-test-slave-2j7r krb5kdc[27171](info): commencing operation
krb5kdc: starting...
W20260810 19:31:37.300983 19728 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/test-user.keytab.
Aug 10 19:31:37 dist-test-slave-2j7r krb5kdc[27171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
Entry for principal kudu/127.19.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab.
I20260810 19:31:37.629675 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35931
--webserver_interface=127.19.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:31:37.974831 27199 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:37.975138 27199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:37.975238 27199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:37.975358 27199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:37.982502 27199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:37.982587 27199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:37.982653 27199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:37.982728 27199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:37.982815 27199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:37.992532 27199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.62.keytab
--principal=kudu/127.19.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.62:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:37.993917 27199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:37.995194 27199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:38.006841 27211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:38.006692 27213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:38.006878 27208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:38.007489 27199 server_base.cc:1034] running on GCE node
I20260810 19:31:38.008378 27199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:38.009989 27199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:38.011278 27199 hybrid_clock.cc:648] HybridClock initialized: now 1786390298011166 us; error 44 us; skew 500 ppm
I20260810 19:31:38.011646 27199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:38.014840 27199 init.cc:377] Logged in from keytab as kudu/127.19.68.62@KRBTEST.COM (short username kudu)
I20260810 19:31:38.016218 27199 webserver.cc:460] Webserver started at http://127.19.68.62:38793/ using document root <none> and password file <none>
I20260810 19:31:38.016888 27199 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:38.016993 27199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:38.017336 27199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:38.020309 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data/instance:
uuid: "9fc6a206b40740e393fa52c011b67155"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "6f6cf656a114f2dd64c2e0e64ac50684"
server_key_iv: "ee5c54bb14185d2ac340fdeaeb810726"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.021116 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance:
uuid: "9fc6a206b40740e393fa52c011b67155"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "6f6cf656a114f2dd64c2e0e64ac50684"
server_key_iv: "ee5c54bb14185d2ac340fdeaeb810726"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.026171 27199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:31:38.029471 27227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:38.030707 27199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:31:38.030871 27199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal
uuid: "9fc6a206b40740e393fa52c011b67155"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "6f6cf656a114f2dd64c2e0e64ac50684"
server_key_iv: "ee5c54bb14185d2ac340fdeaeb810726"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.031071 27199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:38.048998 27199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:38.052644 27199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:38.052904 27199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:38.070456 27199 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.62:35931
I20260810 19:31:38.070477 27313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.62:35931 every 8 connection(s)
I20260810 19:31:38.071779 27199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/data/info.pb
I20260810 19:31:38.075145 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27199
I20260810 19:31:38.075317 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-0/wal/instance
I20260810 19:31:38.075518 27314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:38.075811 19728 external_mini_cluster.cc:1428] Setting key 4546dc7c8b3ed8f74ee8cacc60ef2cae
I20260810 19:31:38.081848 27314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:38.104692 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.084770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35931 (local address 127.0.0.1:59236)
0810 19:31:38.085933 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.085991 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:38.088100 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:38.088907 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:38.088913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:38.089670 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:38.090106 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.090131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.091827 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.091831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.092891 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.092908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.094107 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.094112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.094248 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.094900 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:38.094921 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:38.097228 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:38.100959 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.100969 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.100978 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.101423 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.101858 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.101861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.101863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.102016 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.102872 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:38.102878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:38.103070 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 19:31:38.103266 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 19:31:38.103656 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":708,"threads_started":1}
I20260810 19:31:38.104719 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.084771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.19.68.62:35931)
0810 19:31:38.085609 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:38.085625 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:38.086126 (+ 501us) server_negotiation.cc:407] Connection header received
0810 19:31:38.088447 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:38.088453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:38.088676 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:38.088826 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:38.090375 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.091655 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.093034 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.093862 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.094018 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.097462 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:38.097471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:38.097485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:38.097519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:38.100829 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.101559 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.101567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.101570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.101751 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.102135 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.102139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.102142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.102760 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:38.102955 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:38.103470 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 19:31:38.103834 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260810 19:31:38.107563 27314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:38.108737 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.108301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55202)
0810 19:31:38.108512 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.108635 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 19:31:38.108994 27232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
I20260810 19:31:38.110518 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.110189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55204)
0810 19:31:38.110327 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.110392 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:31:38.110697 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:46091: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
Entry for principal kudu/127.19.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab.
I20260810 19:31:38.149905 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:46091
--webserver_interface=127.19.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:31:38.160025 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } attempt: 1
I20260810 19:31:38.161190 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.160719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55212)
0810 19:31:38.160918 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.161008 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:31:38.162190 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.161811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55218)
0810 19:31:38.161952 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.162027 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:31:38.162386 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:46091: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
I20260810 19:31:38.230628 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } attempt: 2
I20260810 19:31:38.231769 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.231313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55226)
0810 19:31:38.231526 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.231618 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:31:38.232921 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.232535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55230)
0810 19:31:38.232684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.232778 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 19:31:38.233153 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:46091: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
I20260810 19:31:38.324411 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } attempt: 3
I20260810 19:31:38.325723 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.325241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55244)
0810 19:31:38.325450 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.325552 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:31:38.326855 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.326517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55258)
0810 19:31:38.326644 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.326715 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:31:38.327064 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:46091: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
I20260810 19:31:38.470306 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } attempt: 4
I20260810 19:31:38.471552 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.470983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55260)
0810 19:31:38.471321 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.471412 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260810 19:31:38.472587 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.472173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55262)
0810 19:31:38.472360 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.472448 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 19:31:38.472798 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.61:46091: Network error: Client connection negotiation failed: client connection to 127.19.68.61:46091: connect: Connection refused (error 111)
W20260810 19:31:38.477638 27323 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:38.477948 27323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:38.478041 27323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:38.478158 27323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:38.485253 27323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:38.485339 27323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:38.485450 27323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:38.485517 27323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:38.485601 27323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:38.495319 27323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.61.keytab
--principal=kudu/127.19.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.61:46091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:38.496589 27323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:38.497905 27323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:31:38.509238 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:38.509363 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:38.509665 27341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:38.510090 27323 server_base.cc:1034] running on GCE node
I20260810 19:31:38.510763 27323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:38.512014 27323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:38.513190 27323 hybrid_clock.cc:648] HybridClock initialized: now 1786390298513125 us; error 37 us; skew 500 ppm
I20260810 19:31:38.513496 27323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:38.520100 27323 init.cc:377] Logged in from keytab as kudu/127.19.68.61@KRBTEST.COM (short username kudu)
I20260810 19:31:38.521577 27323 webserver.cc:460] Webserver started at http://127.19.68.61:41497/ using document root <none> and password file <none>
I20260810 19:31:38.522253 27323 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:38.522358 27323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:38.522665 27323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:38.525646 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data/instance:
uuid: "30860848d48a423bbfdb3155eb44173a"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "9326edc8ded7dddc019e50b1ff89154a"
server_key_iv: "8c2c9a0e2cb8c2f794c86b15c9fef9d0"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.526440 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal/instance:
uuid: "30860848d48a423bbfdb3155eb44173a"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "9326edc8ded7dddc019e50b1ff89154a"
server_key_iv: "8c2c9a0e2cb8c2f794c86b15c9fef9d0"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.531466 27323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:31:38.534853 27353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:38.536162 27323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:31:38.536326 27323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal
uuid: "30860848d48a423bbfdb3155eb44173a"
format_stamp: "Formatted at 2026-08-10 19:31:38 on dist-test-slave-2j7r"
server_key: "9326edc8ded7dddc019e50b1ff89154a"
server_key_iv: "8c2c9a0e2cb8c2f794c86b15c9fef9d0"
server_key_version: "encryptionkey@0"
I20260810 19:31:38.536521 27323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:38.549744 27323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:38.553897 27323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:38.554165 27323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:38.571806 27323 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.61:46091
I20260810 19:31:38.571846 27453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.61:46091 every 8 connection(s)
I20260810 19:31:38.573233 27323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/data/info.pb
I20260810 19:31:38.576959 27454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:38.579689 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27323
I20260810 19:31:38.579892 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-1/wal/instance
I20260810 19:31:38.580461 19728 external_mini_cluster.cc:1428] Setting key b90cc7e2f4fdf7f62bb47a9bd5a33f60
I20260810 19:31:38.585167 27454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:38.605609 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.588113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.19.68.62:35931)
0810 19:31:38.588332 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:38.588339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:38.589561 (+ 1222us) server_negotiation.cc:407] Connection header received
0810 19:31:38.591369 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:38.591375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:38.591457 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:38.591563 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:38.592951 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.593687 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.595615 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.596029 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.596117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.599661 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:38.599669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:38.599674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:38.599721 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:38.602763 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.603434 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.603439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.603442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.603586 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.603935 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.603939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.603942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.604286 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:38.604455 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:38.605067 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 19:31:38.605312 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:31:38.606353 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.588563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35931 (local address 127.0.0.1:59238)
0810 19:31:38.589372 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.589442 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:38.591107 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:38.591745 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:38.591753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:38.592251 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:38.592783 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.592799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.593851 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.593857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.595498 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.595513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.596237 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.596243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.596513 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.597205 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:38.597231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:38.599403 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:38.602888 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.602894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.602905 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.603317 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.603676 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.603679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.603682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.603837 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.604397 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:38.604403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:38.604718 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 19:31:38.604928 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 19:31:38.605312 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":345,"threads_started":1}
I20260810 19:31:38.608055 27454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.68.60@KRBTEST.COM; defaulting to no policy
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.68.60@KRBTEST.COM" created.
I20260810 19:31:38.622496 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.608803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55268)
0810 19:31:38.608962 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.608979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:38.609071 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:38.610148 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:38.610157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:38.610575 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:38.610739 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.610749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.612051 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.612057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.613372 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.613388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.614177 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.614186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.614352 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.614882 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:38.614905 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:38.617203 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:38.620282 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.620285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.620288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.620624 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.621032 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.621036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.621039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.621198 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.621844 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:38.621847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:38.621957 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 19:31:38.622056 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:31:38.622236 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:31:38.622829 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.609004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.19.68.61:46091)
0810 19:31:38.609657 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:38.609663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:38.609687 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:31:38.609766 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:38.609772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:38.609993 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:38.610100 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:38.610898 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.611908 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.613510 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.614024 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.614115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.617381 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:38.617392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:38.617403 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:38.617431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:38.620142 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.620760 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.620768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.620772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.620941 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.621305 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.621310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.621314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.621720 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:38.621910 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:38.622254 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 19:31:38.622491 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":364,"threads_started":1}
I20260810 19:31:38.625073 27454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:38.626188 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.625775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58242)
0810 19:31:38.625936 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.626063 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 19:31:38.626443 27357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.627836 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.627530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58244)
0810 19:31:38.627664 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.627736 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:31:38.628036 27454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
Entry for principal kudu/127.19.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab.
I20260810 19:31:38.655965 19728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
/tmp/dist-test-taskTNmuTH/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:41595
--webserver_interface=127.19.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.68.20:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:31:38.677377 27454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 1
I20260810 19:31:38.678740 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.678160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58254)
0810 19:31:38.678401 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.678550 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 19:31:38.679790 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.679414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58266)
0810 19:31:38.679577 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.679665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 19:31:38.679996 27454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.748250 27454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 2
I20260810 19:31:38.749490 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.748899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58282)
0810 19:31:38.749148 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.749264 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 19:31:38.750550 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.750135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58290)
0810 19:31:38.750287 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.750396 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:31:38.750831 27454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.772063 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } attempt: 5
Aug 10 19:31:38 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:38.786124 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.772900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.19.68.61:46091)
0810 19:31:38.773114 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:38.773120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:38.773143 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:31:38.773301 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:38.773304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:38.773387 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:38.773485 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:38.774471 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.775418 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.777149 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.777596 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.777691 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.781209 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:38.781213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:38.781217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:38.781247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:38.783644 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.784384 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.784388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.784390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.784508 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:38.784861 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:38.784864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:38.784866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:38.785152 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:38.785262 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:38.785724 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 19:31:38.785881 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:31:38.786271 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.772779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55274)
0810 19:31:38.772997 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.773018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:38.773181 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:38.773537 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:38.773542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:38.774066 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:38.774305 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.774318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.775575 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.775581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.776944 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:38.776962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:38.777759 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:38.777765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:38.777977 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:38.778586 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:38.778609 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:38.781015 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:38.783785 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.783790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.783794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.784279 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.784601 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:38.784606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:38.784609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:38.784773 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:38.785257 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:38.785262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:38.785460 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 19:31:38.785654 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 19:31:38.785922 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:31:38.787388 27314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:31:38.788542 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.788060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58294)
0810 19:31:38.788245 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.788358 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:31:38.789553 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.789164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58304)
0810 19:31:38.789304 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.789412 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 19:31:38.789781 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.841012 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 1
I20260810 19:31:38.842054 27454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 3
I20260810 19:31:38.842267 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.841702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58306)
0810 19:31:38.841920 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.842059 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:31:38.843106 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.842580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58322)
0810 19:31:38.842816 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.842917 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 19:31:38.843359 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.842975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58330)
0810 19:31:38.843131 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.843219 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:31:38.843614 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.844274 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.843883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58342)
0810 19:31:38.844099 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.844171 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260810 19:31:38.844496 27454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.911818 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 2
I20260810 19:31:38.913041 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.912552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58346)
0810 19:31:38.912763 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.912877 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:31:38.914104 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.913809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58362)
0810 19:31:38.913925 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.914002 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 19:31:38.914311 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
I20260810 19:31:38.987743 27454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 4
I20260810 19:31:38.988864 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.988393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58368)
0810 19:31:38.988620 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.988702 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 19:31:38.989813 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:38.989490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58370)
0810 19:31:38.989640 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:38.989707 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:31:38.990020 27454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
W20260810 19:31:38.993510 27464 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:31:38.993813 27464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:31:38.993911 27464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:31:38.994019 27464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:31:39.001145 27464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:31:39.001237 27464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:31:39.001335 27464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:31:39.001400 27464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:31:39.001482 27464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:31:39.011332 27464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.68.20:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.68.62:35931,127.19.68.61:46091,127.19.68.60:41595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/krb5kdc/kudu_127.19.68.60.keytab
--principal=kudu/127.19.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.68.60:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:31:39.012653 27464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:31:39.014075 27464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:31:39.020493 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 3
I20260810 19:31:39.021593 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.021156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58374)
0810 19:31:39.021340 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.021429 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:31:39.022564 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.022192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58378)
0810 19:31:39.022387 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.022456 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 19:31:39.022751 27314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.68.60:41595: Network error: Client connection negotiation failed: client connection to 127.19.68.60:41595: connect: Connection refused (error 111)
W20260810 19:31:39.025579 27478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:39.025624 27470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:31:39.025647 27472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:31:39.026443 27464 server_base.cc:1034] running on GCE node
I20260810 19:31:39.027189 27464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:31:39.028505 27464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:31:39.029721 27464 hybrid_clock.cc:648] HybridClock initialized: now 1786390299029659 us; error 34 us; skew 500 ppm
I20260810 19:31:39.030033 27464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:31:39.032960 27464 init.cc:377] Logged in from keytab as kudu/127.19.68.60@KRBTEST.COM (short username kudu)
I20260810 19:31:39.034556 27464 webserver.cc:460] Webserver started at http://127.19.68.60:38191/ using document root <none> and password file <none>
I20260810 19:31:39.035413 27464 fs_manager.cc:359] Metadata directory not provided
I20260810 19:31:39.035543 27464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:31:39.035861 27464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:31:39.038856 27464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data/instance:
uuid: "4ea57db19b94460cbfc6157a16d19692"
format_stamp: "Formatted at 2026-08-10 19:31:39 on dist-test-slave-2j7r"
server_key: "3551398ba22de4f93f0b579b34a76ae7"
server_key_iv: "8b9b285d55bb9a5b9ae0122cabfc1d36"
server_key_version: "encryptionkey@0"
I20260810 19:31:39.039661 27464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal/instance:
uuid: "4ea57db19b94460cbfc6157a16d19692"
format_stamp: "Formatted at 2026-08-10 19:31:39 on dist-test-slave-2j7r"
server_key: "3551398ba22de4f93f0b579b34a76ae7"
server_key_iv: "8b9b285d55bb9a5b9ae0122cabfc1d36"
server_key_version: "encryptionkey@0"
I20260810 19:31:39.044680 27464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:31:39.047977 27490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:31:39.049235 27464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:31:39.049377 27464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data,/tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal
uuid: "4ea57db19b94460cbfc6157a16d19692"
format_stamp: "Formatted at 2026-08-10 19:31:39 on dist-test-slave-2j7r"
server_key: "3551398ba22de4f93f0b579b34a76ae7"
server_key_iv: "8b9b285d55bb9a5b9ae0122cabfc1d36"
server_key_version: "encryptionkey@0"
I20260810 19:31:39.049580 27464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:31:39.070787 27464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:31:39.075734 27464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:31:39.076010 27464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:31:39.093941 27464 rpc_server.cc:307] RPC server started. Bound to: 127.19.68.60:41595
I20260810 19:31:39.093976 27589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.68.60:41595 every 8 connection(s)
I20260810 19:31:39.095188 27464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/data/info.pb
I20260810 19:31:39.098278 19728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu as pid 27464
I20260810 19:31:39.098496 19728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTNmuTH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389929706493-19728-0/minicluster-data/master-2/wal/instance
I20260810 19:31:39.099023 27590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:31:39.099102 19728 external_mini_cluster.cc:1428] Setting key 1f7b13a18807ced315217db11e8d40cd
I20260810 19:31:39.108299 27590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
I20260810 19:31:39.122373 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.100894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.19.68.60:41595)
0810 19:31:39.101658 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.101670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.101709 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:31:39.103633 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.103653 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.104046 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.104444 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.106228 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.107490 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.109310 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.109990 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.110121 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.114644 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.114650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.114660 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.114695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.118104 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.118771 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.118779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.118787 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.119002 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.119376 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.119380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.119382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.119911 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.120166 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.120610 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.121040 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":353,"threads_started":1}
I20260810 19:31:39.126860 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.111190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.62:35931 (local address 127.0.0.1:59250)
0810 19:31:39.112108 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.112148 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:39.112297 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:39.112655 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:39.112659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:39.113286 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:39.113553 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.113563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.114951 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.114960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.116594 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.116610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.117373 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.117380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.117569 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.118123 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:39.118145 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:39.120462 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:39.124124 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.124135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.124142 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.124535 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.124936 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.124940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.124942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.125123 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.125843 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:39.125849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:39.126024 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 19:31:39.126178 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 19:31:39.126477 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":521,"threads_started":1}
I20260810 19:31:39.126924 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.111197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.19.68.62:35931)
0810 19:31:39.111719 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.111726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.112237 (+ 511us) server_negotiation.cc:407] Connection header received
0810 19:31:39.112416 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.112421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.112502 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.112619 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.113704 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.114785 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.116733 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.117189 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.117284 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.120658 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.120665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.120669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.120704 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.124002 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.124651 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.124657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.124661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.124821 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.125234 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.125239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.125242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.125620 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.125777 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.126327 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.126552 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":285,"threads_started":1}
I20260810 19:31:39.128289 27590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.62@KRBTEST.COM
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:39.140399 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.125396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.19.68.62:35931)
0810 19:31:39.125984 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.125988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.126007 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:31:39.126044 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.126047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.126096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.126184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.127540 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.128399 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.129968 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.130402 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.130513 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.134736 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.134743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.134747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.134788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.137721 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.138482 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.138486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.138488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.138596 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.138949 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.138953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.138956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.139304 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.139580 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.139852 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.140081 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":263,"threads_started":1}
I20260810 19:31:39.142031 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.128970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.61:46091 (local address 127.0.0.1:55284)
0810 19:31:39.129150 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.129168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:39.129242 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:39.129495 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:39.129498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:39.129802 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:39.129938 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.129945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.130896 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.130901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.131871 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.131884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.132580 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.132586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.132759 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.133294 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:39.133317 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:39.135760 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:39.138939 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.138944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.138948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.139436 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.139806 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.139811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.139815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.140005 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.141196 (+ 1191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:39.141200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:39.141359 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 19:31:39.141497 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 19:31:39.141707 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:31:39.142207 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.129024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.19.68.61:46091)
0810 19:31:39.129232 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.129237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.129259 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:31:39.129315 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.129320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.129390 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.129492 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.130088 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.130748 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.132023 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.132439 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.132546 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.135975 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.135980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.135983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.136017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.138785 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.139537 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.139543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.139546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.139701 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.140339 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.140343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.140345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.141015 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.141195 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.141627 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.141828 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 19:31:39.143328 27590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.60@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.68.61@KRBTEST.COM
I20260810 19:31:39.156379 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.144027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58396)
0810 19:31:39.144228 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.144243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:39.144339 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:39.144597 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:39.144601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:39.144961 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:39.145137 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.145148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.146147 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.146150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.146954 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.146964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.147634 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.147640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.147834 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.148394 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:39.148411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:39.150448 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:39.153878 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.153883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.153887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.154365 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.154692 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.154697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.154701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.154874 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.155404 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:39.155408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:39.155534 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 19:31:39.155688 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 19:31:39.155985 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 19:31:39.156683 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.144017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.19.68.60:41595)
0810 19:31:39.144173 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.144178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.144302 (+ 124us) server_negotiation.cc:407] Connection header received
0810 19:31:39.144431 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.144435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.144493 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.144566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.145280 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.146026 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.147078 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.147472 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.147579 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.150694 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.150701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.150706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.150747 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.153741 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.154477 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.154481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.154484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.154591 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.154972 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.154975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.154977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.155288 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.155432 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.155910 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.156285 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:31:39.157930 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.142230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.19.68.61:46091)
0810 19:31:39.142776 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.142782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.142801 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:31:39.142866 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.142871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.142947 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.143039 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.144833 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.145812 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.147004 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.147423 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.147533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.152131 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.152136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.152139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.152169 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.155087 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.155810 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.155815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.155818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.155975 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.156505 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.156510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.156514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.156851 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.157111 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.157426 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.157622 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":2980,"threads_started":1}
I20260810 19:31:39.162055 27590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: Bootstrap starting.
I20260810 19:31:39.162688 19728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:31:39.165302 27590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:39.166326 27590 log.cc:824] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:39.165060 27270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9fc6a206b40740e393fa52c011b67155"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59252
I20260810 19:31:39.169039 27590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: No bootstrap required, opened a new log
I20260810 19:31:39.179558 27590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.179947 27590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea57db19b94460cbfc6157a16d19692, State: Initialized, Role: FOLLOWER
I20260810 19:31:39.180420 27590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.181413 27605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.181782 27605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.182554 27590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:31:39.191461 27624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:31:39.191633 27624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:31:39.189574 27464 master_runner.cc:186] Error getting master registration for 127.19.68.62:35931: OK, instance_id { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" instance_seqno: 1786390298065426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:31:39.197234 27464 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:31:39.199941 27314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 4
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.62@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
I20260810 19:31:39.217072 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.200613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58406)
0810 19:31:39.200809 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.200827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:39.200930 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:39.201307 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:39.201313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:39.201846 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:39.202056 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.202068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.203647 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.203654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.205203 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.205220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.206308 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.206314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.206540 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.207210 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:39.207233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:39.210086 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:39.214246 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.214253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.214257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.214771 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.215174 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.215178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.215182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.215352 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.215997 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:39.216002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:39.216200 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 19:31:39.216354 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 19:31:39.216684 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:31:39.217160 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.200664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.19.68.60:41595)
0810 19:31:39.200936 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.200942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.200964 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:31:39.201024 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.201032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.201137 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.201244 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.202224 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.203466 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.205360 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.206155 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.206302 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.210312 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.210319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.210325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.210371 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.214099 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.214902 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.214909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.214912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.215067 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.215464 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.215469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.215472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.215884 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.216031 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.216490 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.216729 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260810 19:31:39.222556 27314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: Bootstrap starting.
I20260810 19:31:39.225811 27314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:39.226902 27314 log.cc:824] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:39.229717 27314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: No bootstrap required, opened a new log
I20260810 19:31:39.235605 27314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.235948 27314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc6a206b40740e393fa52c011b67155, State: Initialized, Role: FOLLOWER
I20260810 19:31:39.236395 27314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.237377 27630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.237787 27630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.238556 27314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:39.238687 27270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:31:39.238912 27270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:39.242703 27270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.245268 27542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fc6a206b40740e393fa52c011b67155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ea57db19b94460cbfc6157a16d19692"
I20260810 19:31:39.245808 27542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:39.249377 27542 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc6a206b40740e393fa52c011b67155 in term 1.
I20260810 19:31:39.250094 27228 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea57db19b94460cbfc6157a16d19692, 9fc6a206b40740e393fa52c011b67155; no voters:
I20260810 19:31:39.250600 27630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:31:39.251403 27630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc6a206b40740e393fa52c011b67155, State: Running, Role: LEADER
I20260810 19:31:39.251974 27630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.255707 27655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc6a206b40740e393fa52c011b67155. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.256145 27655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:39.256544 27270 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [CANDIDATE]: Term 1 election: Requested vote from peers 30860848d48a423bbfdb3155eb44173a (127.19.68.61:46091), 4ea57db19b94460cbfc6157a16d19692 (127.19.68.60:41595)
I20260810 19:31:39.265702 27657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:31:39.278690 27657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:31:39.289208 27454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } attempt: 5
Aug 10 19:31:39 dist-test-slave-2j7r krb5kdc[27171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786390298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.68.61@KRBTEST.COM for kudu/127.19.68.60@KRBTEST.COM
W20260810 19:31:39.305570 27238 tablet.cc:2369] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:31:39.312927 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.289876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.68.60:41595 (local address 127.0.0.1:58422)
0810 19:31:39.290263 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 19:31:39.290284 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:31:39.290409 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:31:39.291027 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:31:39.291033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:31:39.291592 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:31:39.291809 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.291823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.293444 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.293450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.294856 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:31:39.294873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.296128 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.296135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:31:39.296381 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.297145 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:31:39.297192 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:31:39.301043 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:31:39.306693 (+ 5650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.306699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.306703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.307609 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.308349 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:31:39.308356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:31:39.308360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:31:39.308686 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:31:39.309730 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:31:39.309735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:31:39.309982 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 19:31:39.310277 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 19:31:39.311296 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260810 19:31:39.313825 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.290282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.19.68.60:41595)
0810 19:31:39.290489 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.290495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.290517 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:31:39.290584 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.290589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.290672 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.290782 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.292036 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.293212 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.295088 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.295755 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.295883 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.301328 (+ 5445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.301336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.301341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.301383 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.306499 (+ 5116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.307786 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.307793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.307797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.308164 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.308838 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.308871 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.308875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.309528 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.310433 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.310935 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.311747 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 19:31:39.320019 27454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: Bootstrap starting.
I20260810 19:31:39.324358 27454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: Neither blocks nor log segments found. Creating new log.
I20260810 19:31:39.327913 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.283743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.19.68.62:35931)
0810 19:31:39.283926 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.283932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.291063 (+ 7131us) server_negotiation.cc:407] Connection header received
0810 19:31:39.293414 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.293420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.293533 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.293601 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.301216 (+ 7615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.302354 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.304300 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.305004 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.305126 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.308819 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.308826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.308832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.308874 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.312615 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.325226 (+ 12611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.325244 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.325247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.325452 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.326002 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.326008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.326011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.326421 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.326677 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.327288 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.327550 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 19:31:39.328431 27505 tablet.cc:2369] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:31:39.329787 27454 log.cc:824] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:31:39.278901 27199 master_runner.cc:186] Error getting master registration for 127.19.68.61:46091: OK, instance_id { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" instance_seqno: 1786390298566841 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:31:39.333815 27199 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:31:39.344063 27454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: No bootstrap required, opened a new log
I20260810 19:31:39.349282 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.285410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.19.68.61:46091)
0810 19:31:39.285581 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.285586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.322390 (+ 36804us) server_negotiation.cc:407] Connection header received
0810 19:31:39.324240 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.324246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.324327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.324394 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.330757 (+ 6363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.331879 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.336912 (+ 5033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.337615 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.337711 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.341941 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.341948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.341952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.341991 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.345738 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.346870 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.346876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.346880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.347054 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.347474 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.347479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.347483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.347867 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.348105 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.348652 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.348898 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:31:39.356299 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.292623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.19.68.60:41595)
0810 19:31:39.322847 (+ 30224us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.322853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.323531 (+ 678us) server_negotiation.cc:407] Connection header received
0810 19:31:39.323594 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.323599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.323676 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.323784 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.327831 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.330552 (+ 2721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.335855 (+ 5303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.336557 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.336682 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.343987 (+ 7305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:31:39.343995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:31:39.344000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:31:39.344044 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:31:39.349250 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.350993 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.350999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.351001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.351415 (+ 414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:31:39.352195 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:31:39.352200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:31:39.352203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:31:39.352903 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:31:39.354931 (+ 2028us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.355168 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.355592 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30044,"thread_start_us":28457,"threads_started":1}
I20260810 19:31:39.358599 27542 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [term 1 FOLLOWER]: Refusing update from remote peer 9fc6a206b40740e393fa52c011b67155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:31:39.359859 27655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:31:39.365681 27454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.366111 27454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30860848d48a423bbfdb3155eb44173a, State: Initialized, Role: FOLLOWER
I20260810 19:31:39.366611 27454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } }
I20260810 19:31:39.367812 27676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.368650 27676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.372476 27398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:31:39.376005 27398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [term 1 FOLLOWER]: Refusing update from remote peer 9fc6a206b40740e393fa52c011b67155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:31:39.376636 27400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fc6a206b40740e393fa52c011b67155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30860848d48a423bbfdb3155eb44173a"
I20260810 19:31:39.379967 27400 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc6a206b40740e393fa52c011b67155 in term 1.
I20260810 19:31:39.380722 27655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:31:39.381285 27454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:31:39.386837 27605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc6a206b40740e393fa52c011b67155. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.387151 27605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.395205 27673 mvcc.cc:205] Tried to move back new op lower bound from 7317054666161561600 to 7317054665741578240. Current Snapshot: MvccSnapshot[applied={T|T < 7317054666161561600}]
I20260810 19:31:39.399582 27655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.399871 27655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:39.405572 27675 mvcc.cc:205] Tried to move back new op lower bound from 7317054666161561600 to 7317054665741578240. Current Snapshot: MvccSnapshot[applied={T|T < 7317054666161561600}]
I20260810 19:31:39.407917 27657 catalog_manager.cc:1347] Generated new cluster ID: 56ba9fae17334050afe64b49f4990241
I20260810 19:31:39.408013 27657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:31:39.407837 27676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc6a206b40740e393fa52c011b67155. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.408133 27676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.418464 27630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.418768 27630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155 [sys.catalog]: This master's current role is: LEADER
I20260810 19:31:39.420538 27605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.420769 27605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.425578 27676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc6a206b40740e393fa52c011b67155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc6a206b40740e393fa52c011b67155" member_type: VOTER last_known_addr { host: "127.19.68.62" port: 35931 } } peers { permanent_uuid: "30860848d48a423bbfdb3155eb44173a" member_type: VOTER last_known_addr { host: "127.19.68.61" port: 46091 } } peers { permanent_uuid: "4ea57db19b94460cbfc6157a16d19692" member_type: VOTER last_known_addr { host: "127.19.68.60" port: 41595 } } }
I20260810 19:31:39.425835 27676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:31:39.451426 27657 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:31:39.453505 27657 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:31:39.473533 27657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc6a206b40740e393fa52c011b67155: Generated new TSK 0
I20260810 19:31:39.475638 27657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:31:39.496166 27700 catalog_manager.cc:1259] Loaded cluster ID: 56ba9fae17334050afe64b49f4990241
I20260810 19:31:39.496367 27700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: loading cluster ID for follower catalog manager: success
I20260810 19:31:39.499960 27700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: acquiring CA information for follower catalog manager: success
I20260810 19:31:39.502424 27700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30860848d48a423bbfdb3155eb44173a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 19:31:39.642355 27461 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.619615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.19.68.61:46091)
0810 19:31:39.619866 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.619873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.640171 (+ 20298us) server_negotiation.cc:407] Connection header received
0810 19:31:39.640504 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.640511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.640605 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.640687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.642064 (+ 1377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55312: BlockingRecv error: recv got EOF from 127.0.0.1:55312 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 19:31:39.644904 27598 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.620684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.19.68.62:35931)
0810 19:31:39.620882 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.620887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.643247 (+ 22360us) server_negotiation.cc:407] Connection header received
0810 19:31:39.643511 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.643516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.643602 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.643673 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.644604 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59272: BlockingRecv error: recv got EOF from 127.0.0.1:59272 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 19:31:39.654305 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.623186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.19.68.60:41595)
0810 19:31:39.623384 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.623389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.645819 (+ 22430us) server_negotiation.cc:407] Connection header received
0810 19:31:39.652889 (+ 7070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.652895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.652981 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.653063 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.654074 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58444: BlockingRecv error: recv got EOF from 127.0.0.1:58444 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:31:39.705505 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.691089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.19.68.61:46091)
0810 19:31:39.691294 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.691300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.697621 (+ 6321us) server_negotiation.cc:407] Connection header received
0810 19:31:39.697865 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.697871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.698109 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.698285 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.699027 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.700293 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.702389 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.703049 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.703166 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.703939 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.704609 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.704753 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.705019 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.705147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 19:31:39.755622 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.743530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.19.68.60:41595)
0810 19:31:39.743744 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.743750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.753380 (+ 9630us) server_negotiation.cc:407] Connection header received
0810 19:31:39.753819 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.753827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.753994 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.754087 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.755340 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58450: BlockingRecv error: recv got EOF from 127.0.0.1:58450 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:31:39.808614 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.791068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.19.68.61:46091)
0810 19:31:39.791281 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.791287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.801886 (+ 10599us) server_negotiation.cc:407] Connection header received
0810 19:31:39.802374 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.802380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.802499 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.802565 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.803239 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.804332 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.806362 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.807018 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.807135 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.807701 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.807913 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.808009 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.808223 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.808308 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 19:31:39.818578 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.816203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.19.68.60:41595)
0810 19:31:39.816379 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.816385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.816914 (+ 529us) server_negotiation.cc:407] Connection header received
0810 19:31:39.817119 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.817125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.817226 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.817337 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.818243 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58464: BlockingRecv error: recv got EOF from 127.0.0.1:58464 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:31:39.831439 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.823681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.19.68.61:46091)
0810 19:31:39.823858 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.823864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.825569 (+ 1705us) server_negotiation.cc:407] Connection header received
0810 19:31:39.825772 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.825777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.825864 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.825973 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.826386 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.827526 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.829452 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.830078 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.830213 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.830638 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.830783 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.830868 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.831004 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.831083 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 19:31:39.840155 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.838003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.19.68.60:41595)
0810 19:31:39.838184 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.838191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.838651 (+ 460us) server_negotiation.cc:407] Connection header received
0810 19:31:39.838819 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.838824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.838934 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.839023 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.839919 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58472: BlockingRecv error: recv got EOF from 127.0.0.1:58472 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:31:39.852973 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.847520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.19.68.61:46091)
0810 19:31:39.847756 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.847761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.847985 (+ 224us) server_negotiation.cc:407] Connection header received
0810 19:31:39.848116 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.848121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.848205 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.848299 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.848625 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.849791 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.851084 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.851758 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.851884 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.852218 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.852356 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.852441 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.852651 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.852713 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260810 19:31:39.864111 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.862273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.19.68.60:41595)
0810 19:31:39.862512 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.862518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.862757 (+ 239us) server_negotiation.cc:407] Connection header received
0810 19:31:39.862919 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.862923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.863008 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.863090 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.863902 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58480: BlockingRecv error: recv got EOF from 127.0.0.1:58480 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 19:31:39.881389 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.875868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.19.68.61:46091)
0810 19:31:39.876064 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.876069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.876245 (+ 176us) server_negotiation.cc:407] Connection header received
0810 19:31:39.876396 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.876401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.876488 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.876591 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.877016 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.878142 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.879443 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.880103 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.880239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.880600 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.880769 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.880865 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.880950 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.881027 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 19:31:39.892714 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.890748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.19.68.60:41595)
0810 19:31:39.890951 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.890956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.891287 (+ 331us) server_negotiation.cc:407] Connection header received
0810 19:31:39.891464 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.891468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.891539 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.891633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.892466 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58494: BlockingRecv error: recv got EOF from 127.0.0.1:58494 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 19:31:39.918035 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.912227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.19.68.61:46091)
0810 19:31:39.912419 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.912425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.912703 (+ 278us) server_negotiation.cc:407] Connection header received
0810 19:31:39.912874 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.912880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.912979 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.913111 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.913557 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.914756 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.916090 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.916754 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.916893 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.917328 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.917463 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.917566 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.917653 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.917759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 19:31:39.925776 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.923848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.19.68.60:41595)
0810 19:31:39.924061 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.924067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.924407 (+ 340us) server_negotiation.cc:407] Connection header received
0810 19:31:39.924566 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.924571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.924683 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.924808 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.925537 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58504: BlockingRecv error: recv got EOF from 127.0.0.1:58504 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:31:39.968629 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:39.962958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.19.68.61:46091)
0810 19:31:39.963164 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.963169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.963480 (+ 311us) server_negotiation.cc:407] Connection header received
0810 19:31:39.963664 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.963668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.963759 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.963852 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:39.964261 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.964972 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.966832 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:39.967334 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:39.967448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:39.967938 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:39.968046 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:39.968110 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:39.968243 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 19:31:39.968313 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 19:31:39.976462 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:39.974371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.19.68.60:41595)
0810 19:31:39.974567 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:39.974572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:39.974922 (+ 350us) server_negotiation.cc:407] Connection header received
0810 19:31:39.975077 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:39.975081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:39.975155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:39.975233 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:39.976232 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58512: BlockingRecv error: recv got EOF from 127.0.0.1:58512 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:31:40.048749 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.044178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.19.68.61:46091)
0810 19:31:40.044388 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.044392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.044716 (+ 324us) server_negotiation.cc:407] Connection header received
0810 19:31:40.044884 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.044888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.044961 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.045045 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.045441 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.046170 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.047419 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.047813 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.047902 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.048221 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.048318 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.048383 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.048468 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.048529 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:31:40.055902 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:40.054137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.19.68.60:41595)
0810 19:31:40.054343 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.054350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.054680 (+ 330us) server_negotiation.cc:407] Connection header received
0810 19:31:40.054842 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.054846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.054920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.054993 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:40.055646 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58518: BlockingRecv error: recv got EOF from 127.0.0.1:58518 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:31:40.192267 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.187318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.19.68.61:46091)
0810 19:31:40.187518 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.187522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.187896 (+ 374us) server_negotiation.cc:407] Connection header received
0810 19:31:40.188045 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.188049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.188133 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.188240 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.188611 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.189403 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.190700 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.191124 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.191237 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.191538 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.191646 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.191722 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.191843 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.191925 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:31:40.193936 27624 catalog_manager.cc:1259] Loaded cluster ID: 56ba9fae17334050afe64b49f4990241
I20260810 19:31:40.194046 27624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: loading cluster ID for follower catalog manager: success
I20260810 19:31:40.196794 27624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: acquiring CA information for follower catalog manager: success
I20260810 19:31:40.199159 27624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ea57db19b94460cbfc6157a16d19692: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 19:31:40.200033 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:31:40.197903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.19.68.60:41595)
0810 19:31:40.198127 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.198133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.198563 (+ 430us) server_negotiation.cc:407] Connection header received
0810 19:31:40.198753 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.198757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.198823 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.198901 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:31:40.199806 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58530: BlockingRecv error: recv got EOF from 127.0.0.1:58530 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:31:40.467640 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.462577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.19.68.61:46091)
0810 19:31:40.462822 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.462828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.463070 (+ 242us) server_negotiation.cc:407] Connection header received
0810 19:31:40.463311 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.463316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.463387 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.463477 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.463913 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.464655 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.466016 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.466425 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.466549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.466963 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.467069 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.467146 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.467296 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.467380 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 19:31:40.478642 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.473463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.19.68.60:41595)
0810 19:31:40.473660 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.473666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.473942 (+ 276us) server_negotiation.cc:407] Connection header received
0810 19:31:40.474075 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.474079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.474238 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.474334 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.474676 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.475353 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.476585 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.476958 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.477050 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.477598 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.478114 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.478224 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.478285 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.478355 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:31:40.495894 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.484912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.19.68.61:46091)
0810 19:31:40.485151 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.485157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.489987 (+ 4830us) server_negotiation.cc:407] Connection header received
0810 19:31:40.490057 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.490060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.490124 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.490199 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.491165 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.492159 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.494179 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.494616 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.494738 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.495163 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.495262 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.495329 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.495487 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.495569 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:31:40.495941 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.484919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.19.68.60:41595)
0810 19:31:40.485150 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.485156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.489320 (+ 4164us) server_negotiation.cc:407] Connection header received
0810 19:31:40.489672 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.489678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.489771 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.489858 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.490904 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.491741 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.493831 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.494483 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.494604 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.495182 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.495295 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.495386 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.495557 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.495642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:31:40.497912 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:31:40.484525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.19.68.62:35931)
0810 19:31:40.488501 (+ 3976us) server_negotiation.cc:206] Beginning negotiation
0810 19:31:40.488508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:31:40.489145 (+ 637us) server_negotiation.cc:407] Connection header received
0810 19:31:40.489288 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:31:40.489294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:31:40.489561 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:31:40.489740 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:31:40.490218 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.491339 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.495036 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:31:40.495910 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:31:40.496029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:31:40.496879 (+ 850us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:31:40.497333 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:31:40.497440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:31:40.497557 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 19:31:40.497639 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3825,"thread_start_us":3701,"threads_started":1}
I20260810 19:31:40.501837 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27199
I20260810 19:31:40.503119 27305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:40.613354 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27199
W20260810 19:31:40.641887 27499 connection.cc:535] server connection from 127.0.0.1:58406 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:31:40.642266 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27323
I20260810 19:31:40.643303 27446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:40.753307 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27323
I20260810 19:31:40.781575 19728 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu pid 27464
I20260810 19:31:40.782717 27575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:31:40.891664 19728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTNmuTH/build/asan/bin/kudu with pid 27464
2026-08-10T19:31:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4860 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9481 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (371142 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST